Introduction
Is a very useful program to take a list and run commands against that list. xargs will take a list of arguments, loop through them and run a command against 1 or more arguments, one at a time.
If you like xargs you might want to check out GNU Parallel.
Basic Example
Here is a really straightforward non-destructive example of using xargs to calculate a MD5 hash on every file in the current directory,
ls | xargs -t md5sum
This is how it works,
- -t will show you what xargs is about to execute before it executes it.
- xargs will by default take the output of ls one line at a time and append it to the end of the command
Thanks to the -t the output will be shown on screen,
md5sum planetary.doc ab5970d50d67bcafe5c554387f76534e = planetary.doc md5sum Superman.jpg cdefa50d737dfcf8dc57886ea1a758c4 = Superman.jpg
Substitution to Rename Files
Now let's get more advanced and use -I to allow substitution and explicitly set the location of what xargs receives. First we'll create a some temporary files,
mkdir temp cd temp touch files1 file2 file3 # Creates 3 empty files
Now using xargs we will add the txt extension to each file,
ls | xargs -t -I{} mv {} {}.txt
mv file1 file1.txt
mv file2 file2.txt
mv file3 file3.txt
The -I{} specifies that the results from ls will be placed in the location of the {} called the replacement string. In fact you can use whatever variable name you want instead of {}. For example, changing {} to varX also works,
ls | xargs -t -IvarX md5 varX md5 file1.txt MD5 (file1.txt) = d41d8cd98f00b204e9800998ecf8427e md5 file2.txt MD5 (file2.txt) = d41d8cd98f00b204e9800998ecf8427e md5 file3.txt MD5 (file3.txt) = d41d8cd98f00b204e9800998ecf8427e
One item I don't understand yet is why {} forces arguments to be iterated through one at a time. Also, how would we allow more than one argument? -n2 will not work.
Debugging xargs with echo
The echo command is useful to test and see what xargs will be looping through,
ls | xargs -I{} echo "mv {} {}.txt"
mv file1 file1.txt
mv file2 file2.txt
mv file3 file3.txt
Notice that using echo I omit the -t but you will want to put the -t back when you are actually executing your command.
Dealing with Special Characters
When using xargs it will not work with special characters like apostrophe in file names.
To get around this limitation use the find command's -print0 option in combination with -0 which handles special characters white space, quote marks, backslashes, blanks and/or newlines,
find . -print0 | xargs -0 -I{} echo {}
This is because "ls" produces slightly different output from "find .". Here is an example,
# Data I am working with
ls
11 My Baby's Got To Pay the Rent 1.m4a 6 Habits (Stay High) [Hippie Sabotage Remix] 1.txt
11 My Baby's Got To Pay the Rent 1.txt Tin's file.txt
11 Summertime Sadness 1.m4a hello
11 The Troubles 1.m4a pwd
12 Canoeing (Katie and Alex's Theme) 1.m4a test123
# Apostrophe kills xargs here
xargs: unterminated quote
# -0 by itself does not solve the problem
ls | xargs -0 -I{} echo {}
{}
# Now it works.
find . -print0 | xargs -0 -I{} echo {}
.
./11 My Baby's Got To Pay the Rent 1.m4a
./11 My Baby's Got To Pay the Rent 1.txt
./11 Summertime Sadness 1.m4a
./11 The Troubles 1.m4a
./12 Canoeing (Katie and Alex's Theme) 1.m4a
./6 Habits (Stay High) [Hippie Sabotage Remix] 1.txt
./hello
./pwd
./test123
./Tin's file.txt
# Remove a file to show -0 works with smaller data set,
rm test123
# -0 by itself now works, but making any file names longer or adding back test123 breaks it
ls | xargs -0 -I{} echo {}
{}
Kitchen-iMac:tmp tin.pham$ ls | xargs -0 -I{} echo {}
11 My Baby's Got To Pay the Rent 1.m4a
11 My Baby's Got To Pay the Rent 1.txt
11 Summertime Sadness 1.m4a
11 The Troubles 1.m4a
12 Canoeing (Katie and Alex's Theme) 1.m4a
6 Habits (Stay High) [Hippie Sabotage Remix] 1.txt
Tin's file.txt
hello
pwd
# Show's that find looks different than ls and you want to keep that in mind,
find . -print0 | xargs -0 -I{} echo {}
.
./11 My Baby's Got To Pay the Rent 1.m4a
./11 My Baby's Got To Pay the Rent 1.txt
./11 Summertime Sadness 1.m4a
./11 The Troubles 1.m4a
./12 Canoeing (Katie and Alex's Theme) 1.m4a
./6 Habits (Stay High) [Hippie Sabotage Remix] 1.txt
./hello
./pwd
./Tin's file.txt
Useful Applications of xargs
Search - ...
find . -print0 | xargs -0 -I{} echo {}