When I do new deployment, my kafka triggered func app starts ingesting data from past 7 days. It takes immense amount of time. So, I want to read data and insert to database with the latest data and not from the start.Therefore, I am using autoOffsetReset flag as latest (assuming this parameter fulfills my requirement)My host.json file
{
"version": "2.0",
"extensions": {
"kafka": {
"autoOffsetReset": "latest"
}
},
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.*, 4.0.0)"
}
}
Function.json — EVENT HUB
{
"scriptFile": "main.py",
"bindings": [
{
"type" : "kafkaTrigger",
"direction": "in",
"name" : "kevents",
"protocol" : "SASLSSL",
"password" : "EventHubConnectionString",
"topic" : "message_python",
"authenticationMode" : "PLAIN",
"cardinality" : "MANY",
"dataType": "string",
"consumerGroup" : "$Default",
"username" : "$ConnectionString",
"BrokerList" : "%BrokerList%"
}
]
}
main.py
import logging
import typing
from azure.functions import KafkaEvent
def main(kevents : typing.List[KafkaEvent]):
for event in kevents:
logging.info(event.get_body())
I have tried different host.json settings but I am unable to read latest events when doing new deployments. Instead, it reads past 7 days data.