Have configured a KDS which will receive events from EventBus. In target rule of bus, while configuring the input transformers, have not defined any partition key for KDS. The KDS receives the data from bus.
Challenge is to view the data. None of the starting positions works apart from Timestamp. . Latest & Trim Horizon, the most used ones, do not show any data. Able to see data only when specific timestamp is given after referring the CloudWatch logs. But that too has an issue. Like if data is received at 10:00:00 UTC & then another data chunk is received at 10:25:00 UTC, then when searching with 10:00:00 UTC timestamp shows data only received at that time & not after that.
- Is this expected?
- Usually, in what range does the timestamp retrieves data? Like from timestamp to ‘x’ mins or hours?
Interesting part is, when ingesting data in KDS via Cloudshell, we had to define partition key, without which data would not be sent. So when we did this, the Trim Horizon retrieved the data. So –
- Is partition key really mandatory?
- Will consumers able to see data if partition key is not defined?
So far, I did not see any docs giving clarity on time range that KDS Timestamp fetches the data or why one of the Starting position options fails to retrieve data.