I have been having troubles using the Distributed package. I have one function that returns a 100 x 4 array. This array is produced by averaging over some dimensions of a SharedArray that is constructed with a distributed for, the code looks something like

@everywhere function GenerateMatrix(H::sparsearray, indices::Array{Int}, a::Float64)
   Gavg = SharedArray{Float64}(100, length(indices))
   Z = Array{Float64}(undef, 100, 4)
   G = lu(a*sparse(I, size(H)) - H) #new array, same size as H
   @sync @distributed for i=1:length(indices)
          main_ind = indices[i]
          Gmain = G[:, main_ind]
           ...
          #Other for operations that fill the matrix Gavg
    end
    Z[:,1] = mean(G_avg, dims=2)
    #And other operations up to Z[:,4]
    return Z
 end

This function has no problems whatsover by itself, the problem comes when I try to generate several of these Z arrays and average them over and over, rewriting the same file and saving as checkpoints every factor of 1000, as follows

using DelimitedFiles
function IterateFiles(a::Float64, N)
        for j=1:N
             ...
             #Create H and the respective indices vector
             Z = GenerateMatrix(H, indices, a)
             if j==1
                 writedlm("Z_sample"*string(j), Z)
             else
                Zprev = readdlm("Z_sample"*string(j-1))
                Ztemp = (Zprev *(j-1)+ Z)/j  
                writedlm("Z_sample"*string(j), Ztemp)
                if (j-1)%1000 != 0 #this deletes the previous file unless it corresponds to a checkpoint
                   rm("Z_sample"*string(j-1)) 
                end
        end

This code excutes just fine and produce files as expected, however it stops producing files after (a random number) of several iterations of the function IterateFiles. It does not produce an error message. When I check the workers with ps auxr in the terminal, at the beginning shows that the program is working just fine and the selected number of processors with the distributed are being used, but a bit after ps auxr just displays the original instruction sent but no other workers from the distributed, and after this happens no new files are produced.

I though it was an memory error but that doesn’t seem to be the problem. There’s no error message nor the main instruction to run the code stops, julia -p 8 code.jl still appears as an outgoing work but is no longer producing results.

I have no clue of what can be going on. I’d greatly appreciate any insight or help you can provide to this.