This is how the rsync command works:
cd /var/ftp/pub/rsync/; rsync -azvx dest_server::ftp .
This is how to generate keys and populate them on the server so you can login to it without a password and run a command:
ssh-keygen -t dsa
cat .ssh/id_dsa.pub | ssh user@dest_ip “(cd .ssh && cat – >> authorized_keys2)”
Here’s how to verify that the local rsync files match the remote, then remove the remote rsync files if they are the same (moving from remote to local using rsync to copy and diff/nfs/ssh to remove)
for X in *; do diff -Naurq $X /mnt/nfs/$X && ssh root@dest_ip “(cd /var/ftp/pub && rm $X)”; done
Here’s an improvement to handle filenames w/ spaces:
(IFS=,; for X in `find . -type f -printf “%h/%f,”`; do echo _________$X; cat $X | ssh root@dest “(cd \”`pwd`\” && diff -Naurq – \”$X\” && rm -rf \”$X\” )”; done)
Linux can do some nice snapshot backups.
Just make a nice little script, adjust the date stamp to your liking and run as often as you like. Of course this script can be improved (such as it’s detection of the last rsync run) I’ll take any improvements to the script or I’ll update it as I make changes to my script:
## New rsync style BACKUPDIR=/root/backups/ro mkdir -p $BACKUPDIR DATE=`date +%Y%m%d` # Note if you will be doing snapshots more frequently (than daily) # then you might use the full date stamp of +%Y%m%d%H%M%S LAST=`ls $BACKUPDIR | grep "^[0-9][0-9]*$" | sort | tail -1` if [ -n "$LAST" ]; then rsync -a --relative --link-dest=../$LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE else rsync -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE fi REMOTE_LAST=`ssh root@172.48.0.2 "ls -d /var/storage/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" | tail -1` REMOTE_LAST=`basename $REMOTE_LAST` if [ -n "$REMOTE_LAST" ]; then rsync -e ssh -a --relative --link-dest=../$REMOTE_LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing root@172.48.0.2:/var/storage/$DATE else rsync -e ssh -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing root@172.48.0.2:/var/storage/$DATE fi ## Old Style #DATE=`date +%Y%m%d` #mkdir -p /root/backups/$DATE #cd / #tar -czvf /root/backups/$DATE/$DATE.tgz `find tftpboot | grep -v img$ | grep -v ^tftpboot$` var/ftp/pub/scripts var/ftp/pub/kickstart var/ftp/pub/testing
On the above script I probably have the ssh key thing set up too.
Rsyncing files from behind a firewall without doing extra work:
rsync -e "ssh -X user@mydomain.com ssh -X user@192.168.1.9" -avx --progress --relative --bwlimit=10 :/pictures/20050420 . user@mydomain.com's password:
You will get the above command line prompt for your first ssh password. You will get the X11 popup for each of the subsequent password prompts (you can do multiple hops). This saves me from having to copy the files to my gateway, then locally, then delete from the gateway, not to mention needing extra space on the gateway.
Selective rsync
Using rsync to copy only a certain file type or file extension, or pattern:
Note: Lots of folks on different web pages keep saying you must match ALL the top level directories but that just is not so, you just have to include all directories with a pattern match (+ */):
Contents of include file (or exclude file — it is the same thing either way you go)
+ */
+ *mp3
– *
Rsync command:
rsync -avx –include-from above-file /source /destination
(You can use either of –include-from or –exclude-from when using the + – patterns)
For rsync you have to include before you exclude. Once you match a pattern then pattern matching stops. If you don’t match a pattern then it falls through until it matches or runs out of patterns. In the case above, the last pattern excludes everything that wasn’t matched above. So:
1. Include all directories (*/), so if it’s a directory it automatically adds it and moves on
2. Includes all patterns (only files left since we added all directories above) that end in mp3
3. Excludes everything that didn’t match either of the 2 above includs.
The only possible downside is that you make a copy of the directory tree on the destination, whether those directories contain *mp3 files or not. Oh well, I can live with that. Here’s a way to quickly remove empty directories (note that rmdir doesn’t remove directories that have contents unless you force it – so we will use the “politeness” of rmdir to our advantage):
find /destination -type d | tac | while read; do rmdir “$REPLY”; done
The above should account for filenames with spaces as well as a large quantity of directory names. Here is a shorter way but works on a smaller list of directories names as well as doesn’t handle spaces well. So if you know that the directory names have no spaces in them and you have less than maybe 1000 directories then the following works good:
rmdir `find /destination -type d | tac`
Let us say you want to match mp3 anywhere in the filename and not just the end then it looks like the following will work for your include file (I didn’t fully test the following but it looks like it works good enough):
+ */
+ *mp3*
– *
Then to match the beginning of a filename you could probably use mp3* or /mp3* (untested).
If you want to use rsync to do multiple files on the source part:
rsync -e “ssh -C user@destination” -avx –progress “:/source1 /src2 /src3” /destination
So I’m sure you could combine this with the above –include-from pattern matching to.
???
+ */
+ *mp3*
– *
???
+ */
+ *mp3*
– /*
???
+ /mydir
+ /mydir1/
– /*
http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html