I know Arrow IPC can write with compressors by creating a writer with related parameters.
I created my writer as follows.
CompressionCodec.Factory factory = new CommonsCompressionFactory();
CompressionUtil.CodecType ct = CompressionUtil.CodecType.LZ4_FRAME;
writer = new ArrowFileWriter(
root,
provider,
channel,
null,
new IpcOption(),
factory,
ct,
Optional.of(1));
// writer = new ArrowFileWriter(root, provider, channel);
writer.start();
The program works, but it is extremely slow. It compresses data from about 200M to 20M, with around 20 minutes.
I used to write it fluently with no compressors, so now I doubt whether I utilized it correctly.
I have examined the Java docs of Arrow; however, I found little about my confusion.
I really appreciate any advice or suggestions.