I would like to use the streamGenerateContent method to pass an image/pdf/some other file to Gemini and have it answer a question about a file. The file would be local and not stored on Google CloudStorage.

Currently, in my Python notebook, I am doing the following:

1. Reading in the contents of the file,
2. Encoding them to base64 (which looks like b'<string>' in Python)
3. Decoding to utf-8 ('<string>' in Python)

I am then storing this (along with the text prompt) in a JSON dictionary which I am passing to the Gemini model via an HTTP put request. This approach works fine. However, if I wanted to pass base64 (b'<string>') and essentially skip step 3 above, how would I be able to do this?

Looking at the part of the above documentation which discusses blob (the contents of the file being passed to the model), it says: “If possible send as text rather than raw bytes.” This seems to imply that you can still send in base64, even if it’s not the recommended approach. Here is a code example to illustrate what I mean:

import base64
import requests

with open(filename, 'rb') as f:
    file = base64.b64encode(f.read()).decode('utf-8') # HOW CAN I SKIP THE DECODING STEP?

url     = … # LINK TO streamGenerateContent METHOD WITH GEMINI EXPERIMENTAL MODEL
headers = … # BEARER TOKEN FOR AUTHORIZATION
data    = { …
            "text": "Extract the written instructions from this image.", # TEXT PROMPT
            "inlineData": {
                "mimeType": "image/png", # OR "application/pdf" OR ANY OTHER FILE TYPE
                "data": file # HERE THIS IS A STRING, BUT WHAT IF IT'S IN BASE64?
            },
          }

requests.put(url=url, json=data, headers=headers)

In this example, if I remove the .decode('utf-8'), I get an error saying that the bytes object is not JSON serializable. I also tried the alternative approach of using the data parameter in the requests.put (data=json.dumps(file) instead of json=data), which ultimately gives me a “400 Error: Invalid payload” in the response. Another possibility that I’ve seen is to use mimeType: application/octet-stream, but that doesn’t seem to be listed as a supported type in the documentation above.

Should I be using something other than JSON for this type of request if I would like my data to be in base64? Is what I’m describing even possible? Any advice on this issue would be appreciated.