I am working on a project and I want to send Gemini 1.5 Pro / Flush an .mp4 video file , and get in return scenes (screenshots) from the video. Wondering if it is possible to get the scenes(images) or not.
If not, Maybe I can get a list of scenes occuration(minutes and seconds) and then use a 3rd party app to screenshot the video?
Tried to use something like this: https://cloud.google.com/vertex-ai/generative-ai/docs/samples/generativeaionvertexai-gemini-video-with-audio
But didn’t figure out how to get images in response. (Using Python)
Only saw an example in google docs for code below but its return value in text not screenshot:
import vertexai
from vertexai.generative_models import GenerativeModel, Part
# TODO(developer): Update and un-comment below line
# PROJECT_ID = "your-project-id"
vertexai.init(project=PROJECT_ID, location="us-central1")
model = GenerativeModel("gemini-1.5-flash-002")
video_file_uri = (
"gs://cloud-samples-data/generative-ai/video/behind_the_scenes_pixel.mp4"
)
image_file_uri = "gs://cloud-samples-data/generative-ai/image/a-man-and-a-dog.png"
prompt = """
Watch each frame in the video carefully and answer the questions.
Only base your answers strictly on what information is available in the video attached.
Do not make up any information that is not part of the video and do not be too
verbose, be to the point.
Questions:
- When is the moment in the image happening in the video? Provide a timestamp.
- What is the context of the moment and what does the narrator say about it?
"""
contents = [
Part.from_uri(video_file_uri, mime_type="video/mp4"),
Part.from_uri(image_file_uri, mime_type="image/png"),
prompt,
]
response = model.generate_content(contents)
print(response.text)
# Example response:
# Here are the answers to your questions.
# - **Timestamp:** 0:48
# - **Context and Narration:** A man and his dog are sitting on a sofa
# and taking a selfie. The narrator says that the story is about a blind man
# and his girlfriend and follows them on their journey together and growing closer.
Yuval Maliniak is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1