Requirement:

1. Read multiple parquet files from a location
2. Find invalid records, store them in-memory and write them to a file at the end of the batch
3. Writer writes valid data to the database

Unsure about the pattern to figure out when reader is done reading all the records or if it is the last chunk to be processed so that writer can be notified to write the file with the invalid records in the same step, since total record count will keep varying and not known upfront.

Read through the spring batch docs and other questions on stackover flow but other than the read count in the ExecutionContext or persisting custom state in the context didn’t find anything concrete. Problem is total record count is not known and will keep varying.