Clojure 1.5 includes reducers. They’re currently experimental, but showing great promise.
One of reducers’ features is that they allow parallel fold operations. Sadly, although fold works when given a lazy sequence, it falls back to a sequential reduce.
There’s no theoretical reason why fold couldn’t work in parallel with a lazy sequence, so I decided to see if I could implement it. It turns out to be very easy. I’ve implemented a function called
foldable-seq that takes a lazy sequence and turns it into something that can be folded in parallel. I’ve checked an example program that uses this to count words in a Wikipedia XML dump into GitHub. The code for
foldable-seq is here.
On my 4-core MacBook Pro, the example program runs in approx, 40 seconds without
foldable-seq and 13 seconds with.