I’m writing a c10k-style service and am trying to evaluate Clojure’s performance. Can Clojure agents handle this scale of concurrency with its thread-based agents? Other high performance systems seem to be moving towards async-IO/events/greenlets, albeit at a seemingly higher complexity cost.
Suppose there are 10,000 clients connected, sending messages that should be appended to 1,000 local files–the Clojure service is trying to write to as many files in parallel as it can, while not letting any two separate requests mangle the same single file by writing at the same time.
Clojure agents are extremely elegant conceptually–they would allow separate files to be written independently and asynchronously, while serializing (in the database sense) multiple requests to write to the same file.
My understanding is that agents work by starting a thread for each operation (assume we are IO-bound and using send-off)–so in this case is it correct that it would start 1,000+ threads? Can current-day systems handle this number of threads efficiently? Most of them should be IO-bound and sleeping most of the time, but I presume there would still be a context-switching penalty that is theoretically higher than async-IO/event-based systems (e.g. Erlang, Go, node.js).
If the Clojure solution can handle the performance, it seems like the most elegant thing to code. However if it can’t handle the performance then something like Erlang or Go’s lightweight processes might be preferable, since they are designed to have tens of thousands of them spawned at once, and are only moderately more complex to implement.
Has anyone approached this problem in Clojure or compared to these other platforms? (Thanks for your thoughts!)
1
agents use an expandable thread pool rather than simply starting a thread per task. so the number of active threads will be cc close to the number of active agents. 1000 threads i seems reasonable for a modern computer. you will likely need to increase the max open file descriptors (assuming Linux)
1
As of the upcoming Clojure 1.5, you can use custom ExecutorServices (through set-agent-send-executor! and set-agent-send-off-executor!).
Doing so you’ll be able to cap your number of threads — if needed, 1k mostly sleeping threads doesn’t seem out of reach for the JVM. Beware of the stack size though.