For an algorithm with very little communication between computers (not sensitive to bandwidth limitations) that uses asynchronous communication (not sensitive to network latency), number of CPUs will dominate performance and networking is mostly irrelevant.

For an algorithm with high communication between computers (very sensitive to bandwidth limitations) that uses synchronous communication (very sensitive to network latency), number of CPUs isn’t going to matter much.

In practice, all algorithms fit between these extremes somewhere.

Amdahl’s law applies to these algorithms. It says (paraphrased as I’m lazy) “the total time is the time taken for the part that can’t be done in parallel, plus the part that can be done in parallel divided by the number of CPUs”. This ignores all networking/communication delays.

It would be trivial to amend Amdahl’s law to take into account the networking/communication delays. For example: “the total time is the time taken for the part that can’t be done in parallel, plus networking delays (telling other CPUs what to do), plus the part that can be done in parallel divided by the number of CPUs, plus networking delays (getting results back from other CPUs)”. However note that this example only works for a certain style of networking (request and reply) and won’t work for other cases (e.g. if the work is pipelined).

The problem is that software is typically not one algorithm – software is a collection of many algorithms. The “amended Amdahl’s law” ends up being too complex (as different things can overlap with networking/communication delays; different communication styles can be in use for different pieces; it depends on timing and bandwidth and latency, etc).

At the end of the day; it’s more practical to write a prototype/simulation and measure actual behaviour (and find bottlenecks, and improve/exchange algorithms, until you can’t find a way of improving it further and give up and claim you’ve found an answer).

Mbps is not the only measure to measure network speed, the other arguable more important one is setup time (how long it takes for a pair of CPUs to be ready to send a packet) which is in many cases an order of magnitude larger than straight up bandwidth.

This is the reason why most algorithms try to chunk as much sending as possible

This means that as soon as the packet size times the bandwidth becomes too small compared to the setup time it stops mattering how fast the network is.

The network topography and placement of the various info being calculated is very important to how fast sending can happen, (as each packet blocks the path it is traveling along unless a store and forward method is used).

I’ll have to dredge up my parallel programming course to give more info on this if you want.