I wanted to process a CSV file which might have millions of records in it.
I wanted to process it as its received in chunks. (process the file contents even before the full file uploaded , process in parts)
I was able to analyze how it can be done for the spring-boot-starter-webflux.
but my application is spring-boot-starter-web.
My implementation for the spring-starter-webflux.
@PostMapping("/upload")
public Mono<Void> uploadFile2(@RequestPart("file") FilePart filePart) {
StringBuilder accumulator = new StringBuilder();
return filePart.content()
.doOnNext(dataBuffer -> {
System.out.println("------------------------ Processing started ----------------------------");
})
.map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
return new String(bytes, StandardCharsets.UTF_8);
})
.flatMap(content -> {
accumulator.append(content);
String accumulatedContent = accumulator.toString();
String[] lines = accumulatedContent.split("n");
// Process all complete lines
for (int i = 0; i <= lines.length - 1; i++) {
System.out.println("Processing line: " + lines[i]);
}
// Check if the last character of the accumulated content is a newline
if (accumulatedContent.charAt(accumulatedContent.length() - 1) == 'n') {
// The last line is complete
accumulator.setLength(0); // Clear the accumulator
} else {
// The last line is incomplete, keep it in the accumulator
accumulator.setLength(0);
accumulator.append(lines[lines.length - 1]);
System.out.println("last line incomplete - " + accumulator);
}
return Mono.empty();
})
.then(Mono.defer(() -> {
// Process any remaining content in the accumulator
if (accumulator.length() > 0) {
System.out.println("Processing last line: " + accumulator.toString());
}
return Mono.empty();
}));
}
How can i implement this same functionality in spring-boot-starter-web.
I tried using apache-commons-fileupload.
The implementation for the apache commons-fileupload in spring-boot-starter-web:
@PostMapping("/upload")
public String handleFileUpload(HttpServletRequest request) {
System.out.println("-------------------------- Start ---------------------");
if (!JakartaServletFileUpload.isMultipartContent(request)) {
return "Error: Form must have enctype=multipart/form-data.";
}
JakartaServletFileUpload upload = new JakartaServletFileUpload();
try {
var iter = upload.getItemIterator(request);
while (iter.hasNext()) {
System.out.println("--------------------- CHUNK --------------------");
FileItemInput item = iter.next();
String name = item.getFieldName();
InputStream stream = item.getInputStream();
if (item.isFormField()) {
System.out.println("Form field " + name + " with value " + Streams.asString(stream) + " detected.");
} else {
System.out.println("File field " + name + " with file name " + item.getName() + " detected.");
// Read InputStream into a byte array
byte[] fileBytes = stream.readAllBytes();
// Save the file
File destinationFile = new File("C:\Users\anon\Desktop\output\"
+ new Random().nextInt(100000) + ".csv");
try (FileOutputStream fos = new FileOutputStream(destinationFile)) {
fos.write(fileBytes);
}
// Create a new InputStream from the byte array
try (InputStream newStream = new ByteArrayInputStream(fileBytes)) {
System.out.println(readFromInputStream(newStream));
}
}
}
} catch (IOException e) {
e.printStackTrace();
return "Error: " + e.getMessage();
}
return "File uploaded successfully.";
}
public static String readFromInputStream(InputStream inputStream) {
System.out.println("--------------- processing chunk -------------");
StringBuilder resultStringBuilder = new StringBuilder();
try (BufferedReader br = new BufferedReader(new InputStreamReader(inputStream))) {
String line;
while ((line = br.readLine()) != null) {
resultStringBuilder.append(line).append("n");
}
} catch (IOException e) {
e.printStackTrace();
}
return resultStringBuilder.toString();
}
but this is not receiving the file in chunks.
Any help on how to do it in Spring-boot-starter-web would be helpful.
Thanks in advance.