Xargs is useful to run in parallel. Its parallel processing is very efficient. Read this post about its efficiency and this one about basic commands.

Below, is my favorite way to run with a single-command method repeated (or parallelized):

# not parallel
inputlist | xargs -I{} command

# parallel with 15 threads
inputlist | xargs -P15 -I{} command

Replace inputlist with anything. Each line of input list gets run once by command. In the command you can use {} if you need to call the line.

Below is a multi-command method:

# not parallel
inputlist | xargs -I{} /bin/bash -c 'command1; command2;'

# parallel with 15 threads
inputlist | xargs -P15 -I{} /bin/bash -c 'command1; command2;'

Note: recommend to use single quotes on the outside as seen here

Note: you can redirect the output at the very left outside of the single quotes, this way each run’s output is saved. you can save each seperate run if you redirect inside the quotes.

Ex redirection examples:

# save the output of all into one file:
inputlist | xargs -I{} command
 > output.txt
inputlist | xargs -P5 -I{} command
 > output.txt

# save the output of all into one file:
inputlist | xargs -I{} /bin/bash -c 'command1; command2;'
 > output.txt
inputlist | xargs -P15 -I{} /bin/bash -c 'command1; command2;' > output.txt

# Saving each runs seperate output to a different file (assuming input are filename safe items):
inputlist | xargs -I{} /bin/bash -c '(command1; command2;) > output-{}.txt
'
inputlist | xargs -P15 -I{} /bin/bash -c '(command1; command2;) > output-{}.txt'

The end

3 thoughts on “xargs parallel note to self

  1. Here you will find many articles and also some notes. Some notes are notes to self that I didn t mind sharing. Some articles are copies of other peoples posts, with proper citations included, so that I have easy access to the material from my own domain. Some articles lack good formatting because they were done in the spur of the moment, other don t have good spelling or grammar (and I apologize about that before hand). I always try to get the material out before I cross check everything, because I believe one can still learn even valuable information even if the material isn t in the best format. There are plenty ways to get to an article: Search, Latest Posts Comments, Archives, Calendar. Explore and find your favorite way.

Leave a Reply

Your email address will not be published. Required fields are marked *