The actor model of concurrency seems like it might be a good fit for this.

The Model

In case you’re not familiar with this model, it goes as follows:

Actors are threads that run in a loop. Each actor has a producer-consumer message queue; external code and other actors communicate with an actor by sending a message to it (queuing it in its message queue).

An actor’s thread will block waiting for a message in its message queue, and when one appears the actor will deal with it, then loop back to process or wait for the next message. Repeat.

Note: “Actors” are sometimes called “Agents”, but that term is misapplied. See the comment thread below for more.

Architecture

You could create actors specifically for downloading messages (say one per mailserver/mailbox) and other actors for processing the e-mails once they’ve been downloaded.

Connecting the two you could have a single routing actor that would receive references to the downloaded mail files from the fetching actors and either send each reference to an available processing actor or spin up another one to process it if all the other processing actors were busy. When a processing actor was finished processing, it would send a message to the routing actor saying it was done so the routing actor would know it could send another message to it to process.

I’m betting by this point there’s a library for actors for C++ [UPDATE: See comment by @rwong below]. If all else fails you could try Erlang 😉

I’m not sure how the C++ threading libraries work — whether they map threads to a single core or multiple cores — but if this doesn’t do it for you you could take the same concept and instead of using threads have them be discrete processes and use some sort of message passing framework for the communication.

Edit: I’m betting you’ll have a bottleneck at the network, though, so it might not even make sense to want to occupy all the cores at once (unless the processing takes a loooong time).

Edit: Expanded answer and corrected terminology (Agent -> Actor)