I have the following data in mongoDB
Mongo DB Collection Schema
Document Data
{
"id": "1",
"created_at": "2023-06-05T00:00:00.000+00:00"
}
I am loading this Mongo Data from a NoteBook inside DataBricks this way
dataFrame = spark.read
.format("com.mongodb.spark.sql.DefaultSource")
.option("database", "testdb")
.option("collection", "mycollection")
.option("uri", "mongodb+srv://myuser:[email protected]/testdb")
.option("aggregation.pipeline", """[
{
$match: {
created_at: {
$gte: current_timestamp,
}
},
},
]""").load()
What I am trying to do is that, I want to fetch only the documents that are greater than from current timestamp
But after executing the NoteBook , I see the data belongs to year 2023 is also coming .