Originally published January 7, 2018 @ 3:09 pm
Just some quick syntax for running multiple rsync threads in parallel using find and xargs. In many cases (especially if NFS is involved) this can greatly speed up the copy operation.
threads=`grep -c processor /proc/cpuinfo`; src=/mnt/source/; dest=/mnt/target/ ; rsync -aLP -f"+ */" -f"- *" ${src} ${dest} && (cd ${src} && find . -mount -type f -print0 | xargs -0 -n1 -P${threads} -I% rsync -azP % ${dest}/% )
Another approach is to use the GNU Parallel command. Here’s an example piping output of ls into parallel rsync:
ls /source| egrep -v "something_to_exclude" | parallel --gnu --no-notice -j 8 'banner "{}" ; rsync -alxH --bwlimit=1000 --progress --stats --exclude "something_else_to_exclude" "/source/{}" /target/'
The “-j” option for parallel specifies the number of concurrent threads. For CPU-intensive processes, this number should not exceed the number of available CPU cores minus one (to leave something for the OS). In case with rsync, however, the number of parallel threads can be set well in excess of available CPUs, the primary limiting factors being filesystem and network performance.
Experienced Unix/Linux System Administrator with 20-year background in Systems Analysis, Problem Resolution and Engineering Application Support in a large distributed Unix and Windows server environment. Strong problem determination skills. Good knowledge of networking, remote diagnostic techniques, firewalls and network security. Extensive experience with engineering application and database servers, high-availability systems, high-performance computing clusters, and process automation.
