I have a symmetric matrix of about 150k rows and columns in a polars dataframe. This takes some 100GB RAM, which is fine. When I try to .melt()
the dataset (the polars melt, of course), however, my RAM shows that it uses about 600GB to melt, and after the melt, the dataset is about 400GB, much larger than before. Could someone explain to me why that is, and what I can do against it? It’s not so bad that it uses so much RAM in the process, but I would really like the dataset to be the same size as when it started. Any pointers are appreciated!