Wednesday, January 2, 2013

Simple parallel processing with xargs

We've all been there - looking in a nicely tidied up directory, full of archived data - hundreds of lovely data files all gzipped or (better) bzip2'ed; but now you want to use them and you have to uncompress them all... "if only I could use all n CPUs on my local machine to do this!": enter 'xargs'!

It's as simple as:
> ls *.bz2 | xargs -n 1 -P 6 bunzip2
This will set off bunzip2 on all bz2 files in the current directory.

The '-n 1' flag tells xargs to only provide one argument (file) per command line; the '-P 6' tells xargs how many concurrent processes to run.

No comments:

Post a Comment