I haven’t found a way to do the following yet:

1. Read a very large data set
2. Write the data to an Excel file (avoid JVM out-of-memory, but ideally don’t want to write the whole thing to disk)
3. Upload it in parts (to S3 in my case)

Hoping to do all those piece by piece (read some, generate some, write some, repeat).
But haven’t figured out if I can both avoid memory problems and avoid writing to disk. Something like this could maybe work? :

1. Incrementally read the data using pagination
2. Apache POI Streaming looks like a good way to generate it in memory
3. Use AWS S3 multipart upload to transfer it incrementally

But (2) seems to be a problem:

Apache POI streaming keeps a limited number of rows in memory which is great, but the end result is still flushing the workbook to file.

Has anyone done something like this before? (And is it even possible given that Excel is a binary format?)