R Segmentation Fault in Parallel Process with RestRserve and mcparallel I’m developing a REST API using RestRserve in R to manage trading bots. The API allows starting and stopping bot processes. However, I’m encountering segmentation faults and other memory-related errors when starting and stopping bots multiple times. My goal is baisically to create a REST […]

Is it possible to parallelize all the operations with in separate code chunks. For example, I have 3 code chunks that query separate tables in a database. I then perform some data cleaning. Is it possible to have each code chunk be broken into its own process to be ran on a core at the same time as other code chunks? Essentially have process 1+2+3 going at the same time on separate cores instead of sequentially.

I have a function that takes a raster(image). My main goal is “For each pixel find out how far is it from the edge(low pixel value) in all direction. We count the number of step it takes to reach the edge, add it to values calculated in all the direction and take an average. For that pixel the value now becomes this average value”. I am able to achieve this with the code below”

parLapply is unable to call a function in a custom package using clusterExport(cl, "func").

In scientific computing I often want to run loops indefinitely when looking for a possible counterexample, such as repeat{ if(counterexample)break }. I also want to use parallel::mclapply to take advantage of multiple cores. Is there a way to do this? I would rather not create a huge vector e.g., mclapply(1:1e9, ...) for memory reasons and anyway it isn’t really an endless loop.

I have a large number of RDS files that I need to open, modify and save in place. One iteration of this takes ~1.8s. I have about 40K files that need to be modified, so I attempted to run in parallel. Using 28 processors, it seems like it should take less than an hour to complete, but instead it is taking 4-5x that long. What can be done to fix this? Each file is read by exactly one thread, so there should not be any locking going on. I tried chunking it into blocks of 100 files, but that doesn’t help either. I would expect some overheard from the parallel computations, but this seems way out of line to me.