[Jerman]

Hey guys,

Long time since I have posted her, begging for help! Been learning a lot more about php, and have dont a lot more to my website. Now I have automatic backups going, which is pretty sweet. But that brings me to this question...

I have a shell script that backups my mysql database, my website directory and puts it into a single time stamped zip file. I set crontab to run this once a week. Voila! Automatic backups! Here is the thing... I want to take it a step further so that when the file is done saving, it will log into my server here at home via ssh and upload the file to the server. Not too complicated I would imagine, but I am a bit stumped...

So I am planning on taking this a little further with some ideas I have. But basically I need to be able to also check if a file exists in a directory using a shell script. If it does, then it sends that file. (And then it will move it and stuff, but I know how to do that part of it).

[Jerman]

Well, in some more reading, finally found the find command. I guess all I need to do right now is to figure out the sftp deal, and I will be headed in the right direction.

[danielsza]

I don't know if this is what your looking for but I think it is.

push ssh user@host cat < "local file name" ">" "remote file name"ssh root@192.168.68.2 cat < "/231.gif" ">" "/231.gif"pullssh user@host cat "remote file name" > "local file name"ssh root@192.168.68.2 cat "/231.gif" > "/231.gif"comparessh user@host cat < "remote file name" "|" diff - "local file name"ssh root@192.168.68.2 cat < "/231.gif" "|" diff - "/231.gif"

[AsLan^]

You will probably need a sftp client that allows you to specify a password on the command line.

LFTP is one such client.

I have compiled and tested it on my MacBook so you shouldn't have any problems.

Also, it might not be apparent at first but the syntax you are looking for is

lftp -e "put theFile; quit" -u username,password sftp://remotehost

[Jerman]

Nice, thanks. Going to go look and see what I can do about the sftp deal... I figured there was some way to specify a password with normal sftp in the command. The issue is that being on a shared server, I cannot install any other programs. I have used rsync before in some tests to send afile to my server, perhaps I should do that?

I am revising my idea to where the shell script will run a simple query on my database, to see what files need to be uploaded, and then do it based on that. Removes the need for the searching deal. So I just need to figure out how to "put" the file on my server.

[danielsza]

I wouldn't use a separate sftp client, when you have already have ssh installed. And you can use SSH without entering a password by using private and public keys. Here's a link.

[Jerman]

Oh, so can I copy the files over ssh using the cp command or something? That is the issue, moving the file over.

[danielsza]

you don't use cp to copy files over ssh, you use the command cat, read my post above for the exact command.

[Jerman]

Originally Posted by danielsza you don't use cp to copy files over ssh, you use the command cat, read my post above for the exact command.

Sorry, missed that. I appreciate it. So does the compare part just make sure the files match up? If so, is this done through a hash, or by comparing bit by bit? The limitation for me is my uplink, as it will send at about 150k/sec, which isn't too shabby, but when sending a 50 meg file, it could suck a lot of bandwidth.

*Edit*I am guessing I just need the compare if I wish to have the error checking of course.

[Jerman]

The push method is working great for me! Now to figure out the auto login w/o passwords for ssh. I wish I could just specify a password using -p, would make it a bit easier.

Edit- Auto logins working now. Making good progress.

[colivigan]

Originally Posted by danielsza you don't use cp to copy files over ssh, you use the command cat, read my post above for the exact command.

That's an interesting way to it all right, but I'd just use scp.

[danielsza]

Originally Posted by Jerman Sorry, missed that. I appreciate it. So does the compare part just make sure the files match up? If so, is this done through a hash, or by comparing bit by bit? The limitation for me is my uplink, as it will send at about 150k/sec, which isn't too shabby, but when sending a 50 meg file, it could suck a lot of bandwidth. *Edit*I am guessing I just need the compare if I wish to have the error checking of course.

using the command

ssh user@host cat < "remote file name" "|" diff - "local file name"

makes files be compared on the local host. I'm not 100% sure how it's checked, but i would think it's using a hash.

You could also make the files be compared on the remote host,

ssh user@host cat < "local file name" "|" diff - "remote file name"

*Edit* It looks like it compares the files line for line, not using a hash. You read more about it here.

[Brad]

Originally Posted by UncleJohn That's in interesting way to it all right, but I'd just use scp.

Me too. I'm a big scp user.

Generalized syntax:

scp sourceuser@sourcehost:sourcepath/filename destinationuser@destinationhost:destinationpath/filename

If your destination is simply your local system, you can do:

scp sourceuser@sourcehost:sourcepath/filename destinationpath/filename

[danielsza]

Never having used scp myself, does it support using private/public keys for authentication?

I mostly make my changes using either mc on the server or copying the files using smb or afp.

[chucker]

Originally Posted by Brad Me too. I'm a big scp user. Generalized syntax: scp sourceuser@sourcehost:sourcepath/filename destinationuser@destinationhost:destinationpath/filename If your destination is simply your local system, you can do: scp sourceuser@sourcehost:sourcepath/filename destinationpath/filename

Tangent: I used to use scp, until I discovered that rsync supports the same syntax (as well as the same SSH authentication) and can be quite a bit faster.

[colivigan]

Originally Posted by chucker Tangent: I used to use scp, until I discovered that rsync supports the same syntax (as well as the same SSH authentication) and can be quite a bit faster.

Yes, I was going to mention rsync. IMHO, it may be the most fabulous program ever written. The only potential pitfall is that you can't always count on it being there if you don't have control of the server. Most sysadmins will install it, but not all, and it doesn't come with every OS.

Any system that has ssh, on the other hand, is pretty much guaranteed to support scp.

[chucker]

That's certainly true, but I've been lucky thus far.

[Jerman]

Hmm, what to use... I have done the rsync method before. It would be nice to set up a bit of error checking so that if the file is not sent, it tries again without simply erroring out. How to decide what to use...

(Yeah, my server does support rsync)

[colivigan]

Another point in favor of rsync is that you can run it on entire directories and it will only transfer new or modified files. This would probably help with your error recovery issue. If for some reason a transfer failed, that file would just be picked up next time.

[Jerman]

Originally Posted by UncleJohn Another point in favor of rsync is that you can run it on entire directories and it will only transfer new or modified files. This would probably help with your error recovery issue. If for some reason a transfer failed, that file would just be picked up next time.

Ahh, nice. Can it resume a transferred file that has been interrupted, like on ftp?

[chucker]

Originally Posted by Jerman Ahh, nice. Can it resume a transferred file that has been interrupted, like on ftp?

Yes, with the --partial option (or just use -P, which in addition adds a progress bar).

[Jerman]

Oh, awesome with the progess bar! I will try to move over from cat tonight and try out rsync then. With cat, I am finding that when running via a shell script, it gives no indicator of being done. Right now it looks like:

ssh user@address cat < "backupfile" ">" "targetfile"

Works, but do I need to add an option at the end to disconnect the ssh session or something? I tried adding exit on the next line, but I think it never gets there. I will try this with rsync and see if it presents the same issue.

[colivigan]

The key to getting rsync to use your SSH key authentication is the -e flag to specify the transport. e.g.

rsync -av -e ssh user@host:/remote/path /local/path

or

rsync -av -e ssh /local/path user@host:/remote/path

See here for complete usage, or simply "man rsync" in Terminal.

[chucker]

Originally Posted by UncleJohn The key to getting rsync to use your SSH key authentication is the -e flag to specify the transport. e.g. rsync -av -e ssh user@host:/remote/path /local/path or rsync -av -e ssh /local/path user@host:/remote/path See here for complete usage, or simply "man rsync" in Terminal.

Actually, SSH is the default transport on Mac OS X builds of rsync, or at least has been for a long, long time.

[colivigan]

Originally Posted by chucker Actually, SSH is the default transport on Mac OS X builds of rsync, or at least has been for a long, long time.

Ha! I didn't know that. I've been working on Solaris for so long, as soon as I start typing "rsync" my fingers just sort of add the " -av -e ssh" on their own.

[chucker]

Originally Posted by UncleJohn Ha! I didn't know that. I've been working on Solaris for so long, as soon as I start typing "rsync" my fingers just sort of add the " -av -e ssh" on their own.

From the manpage:

For remote transfers, a modern rsync uses ssh for its communications, but it may have been configured to use a dif- ferent remote shell by default, such as rsh or remsh.

I recall it being rsh somewhere; Cygwin perhaps, or Mac OS X 10.1 or whatever. Perhaps Solaris stuck with it for legacy reasons; changing such defaults tends to break stuff.

[danielsza]

Originally Posted by Jerman Oh, awesome with the progess bar! I will try to move over from cat tonight and try out rsync then. With cat, I am finding that when running via a shell script, it gives no indicator of being done. Right now it looks like: ssh user@address cat < "backupfile" ">" "targetfile" Works, but do I need to add an option at the end to disconnect the ssh session or something? I tried adding exit on the next line, but I think it never gets there. I will try this with rsync and see if it presents the same issue.

No you don't need to disconnect once the file is done copying it will disconnect. Try rsync and see what works best for you.

[Jerman]

All right, everything works great using cat. Will work with rsync and decide if I want to move over to that, but for the time being, it works great as is.

So the next step for me is to learn how to use mysql via the command line. What I need to do is access a database using my shell script. If it finds that a column called "status" equals "download", then it needs to execute my script for getting the file. And it needs to be able to do more than one. So I guess the query needs to return the result as a variable and run a loop. (Which I know in php, but not in bash).

[Gargoyle]

Since every other option has already been mentioned, I'll just throw sftp into the mix.

I am surprised that no-one else suggested it already.

[chucker]

Originally Posted by Gargoyle Since every other option has already been mentioned, I'll just throw sftp into the mix. I am surprised that no-one else suggested it already.

Except, of course, that it has, including in the thread title itself.