I’m in the process of switching from MATLAB to Python. I’m rewriting most of my codes and am using Spyder IDE.

I constantly need to load large datasets of data and plot it. Here I’ve encountered an issue with my use of Python and RAM consumption when plotting the data

Here’s an example of two codes, first in MATLAB and the RAM usage at each point, and then implementation in Python. The goal is to read several HDF5 datasets, stitch them together and plot.

MATLAB:

filename = dir('*00.hdf5');
[~, idx]   = max([filename.datenum]);
filename = filename(idx).name;
dLim = [0 1];
Data = [];
lgth = cellstr(num2str((dLim(1):dLim(2))', '%04d'));
date = filename(1:17);
for i = 1:length(lgth)
    filename1 = [date ,sprintf('.00%s', lgth{i}),'.hdf5'];
    intD = double(h5read(filename1,'/data'));
    Data = [Data; intD];
end

Final size of Data = 58976577×4 RAM usage = 3662 MB
after plot with MATLAB
RAM usage = 7322 MB.

Python:

filenames = glob.glob('*00.hdf5')
filename = max(filenames, key=os.path.getmtime)
filename = os.path.basename(filename)
dLim = [0, 1]
lgth = [f"{i:04d}" for i in range(dLim[0], dLim[1] + 1)]
date = filename[:17]
Data = None
for i in range(len(dLim)):
    filename1 = f"{date}.00{lgth[i]}.hdf5"
    with h5py.File(filename1, 'r') as f:
        intD = np.array(f['/data']).T
        if Data is None:
            Data = intD
        else:
            Data = np.vstack((Data, intD))  

Final size of Data = 58976577×4

RAM usage = 2280 MB

after plotting with matplotlib:

from matplotlib import pyplot as plt

plt.plot(Data)

RAM usage = 11000+ MB.

Summary:
MATLAB RAM consumption:

Final size of Data = 58976577×4

RAM usage = 3662 MB

after plot

RAM usage = 7322 MB

Python (Spyder) RAM consumption:

Final size of Data = 58976577×4

RAM usage = 2280 MB

after plot with matplotlib:

RAM usage = 11000+ MB. Becomes barely responsive.

What am I doing wrong? This is not even the largest dataset that I have worked with before, and I’m stuck… I can downsample the data and reduce the load, but that’s not the point. That can be done in MATLAB as well…