I am processing a file which consist millions of recods. I am sorting the file based on customer id column and creating dynamic batches by making sure that a single customer id record does not falls in two differenct bathces(this is being done because all batches are processed parallely).
As of now I am processing the file through apache camel which reads entire file and returns the file into List. But this approach fails if some larger file comes.