I have a script that is trying to save numpy arrays to an hd5 file one by one. For every array I create a new dataset with the index.

The script was terminated without exit status 0, but since I keep track of how many I am done with I am trying to continue where it left off.

So I open the h5 file in append mode and try to continue writing. This leads to an error

Can't insert link (unable to offset into local heap data block)

Code snippet:

    total = dataset.__len__()
    ckpt_file = output_path + ".ckpt"

    if os.path.exists(output_path):
        with open(ckpt_file, "r") as f:
            continue_from = int(f.read())
            print(f"Continuing from: {continue_from}")

    with h5py.File(output_path, "a") as fout:
        for index in tqdm(range(total)):
            if index <= continue_from:
                continue
            embed = dataset[index]
            embed = embed.to("cpu")
            dset = fout.create_dataset(str(index), (embed.shape[-1]))
            dset[:] = embed.detach().cpu().numpy()
            with open(ckpt_file, "w") as f:
                f.write(str(index))

Any advice is appreciated 🙂

tried looking at whether the file is open or not, the file is definitely closed.