I have an application that uses s3 as storage, I use boto3
to connect to s3. There are around 600,000 files in this storage. I need to search through these files from 1 day ago, and find matches that contain a certain string. I need to perform a query for a timeframe using a date and a string (sha256 hash). How can I perform multiple queries in a single search?
For example:
paginator = client.get_paginator("list_objects")
delta = datetime.datetime.now().replace(tzinfo=pytz.UTC) - datetime.timedelta(timeframe)
filtered_files = paginator.paginate(Bucket="example_bucket", Prefix="my/example/s3/path").search(
"Contents[?to_string(LastModified)>='"{}"' & contains(Key, {})]".format(delta, sha256_hash)
)
The above query search produces the error:
...
ParseError: Expecting: rbracket, got: expref: Parse error at column 72, token "&" (EXPREF), for expression:
"Contents[?to_string(LastModified)>='"time"' & contains(Key, hash)]"