Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
So what is the best way to move something like 314351, files all small but totaling about 20GB? FTP just isn't cutting it. Would rsync be faster or just different? This is from my home system to my dedicated web server.
Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.” Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it. |
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
Can you pack the files together and explode them on the destination?
I'm thinking something along these lines would work: On your Mac: cd /path/to/parent/directory tar -czvf archive.tgz filesdirectory rsync -v archive.tgz remoteuser@remotehost:/path/to/destination/parent/directory On the Linux server: cd /path/to/destination/parent/directory tar -xzvf archive.tgz If you wanted to do it all from your Mac, you could even send the "tar -xzvf" command remotely over ssh. The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
Also, you might want to give the rsync command the arguments "--partial" and "--progress". The first allows you to quickly resume if the transfer gets cut prematurely, and the second gives percentage progress output.
The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
BTW, I suggest packing the files together because that will be much faster than sending them separately. Less handshaking around file starting and ending bits and less IO access as the reads and write skip around the drive.
The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
I had a feeling this was the answer, thanks Brad.
|
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
Working perfectly Brad. Looks like about 6 hours for the upload! I can only imagine what it would have been with FTP and individually.
|
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
Due to server HDD size it looks like I'm going to have to do individual files. It's 16GB as a tarball and 19GB on site expanded. With my other sites on there I just don't have room to copy over from within the server. I'm going to try to set up a sync option instead. Maybe it'll work well this way.
Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.” Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it. |
quote |
http://ga.rgoyle.com
Join Date: May 2004
Location: In your dock hiding behind your finder icon!
|
Why can't you just select all the files in Transmit or something and hit upload (then go for a coffee... or a holiday!) ?
|
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
|
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
That and a holiday is needed for these files and the time it takes. I've gotten rsync to work well for this with the help of Brad. It's going to make my life a whole lot easier.
Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.” Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it. |
quote |
http://ga.rgoyle.com
Join Date: May 2004
Location: In your dock hiding behind your finder icon!
|
I didn't get the feeling that this was needed to be repeatable, more of just a one-off.
|
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
Well, I wasn't sure when I first posted this. I use Transmit which I can do Automator actions for, but rsync is definitely the best option at this point.
Thanks for your help and input anyway. Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.” Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it. |
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
So I've written the sudo code for this, now to actually code it.
Code:
MC Maps Pseudo code
Run backup script on Lenny
Move files backed up from Lenny to Beast <- Local network
Expand files into World RAM folder.
Run map making jar
Rsync results to mcmaps.applenova.com <- Remote network
Replace index.html and specs.js with AN versions <- Remote network How can I do a script that would allow me to automate the remote connection without setting a user to no password? I could create another user and have it only set for accessing that folder, but I'd rather have the same user I'm using now.What options do I have for the cross computer connections? Moving the backup to Beast can be an FTP thing I think, but would rather rsync. So yeah, I'd like rsync for anything that's changing machines, but how do I do it without a password or can I program a password into a script? Sure it's not that secure to have a password in plain text on a script, but these machines aren't web accessible in the locations for the password. Edit: Searching I found this, any thoughts on it? I figure I'll give it a shot. Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.” Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it. Last edited by turtle : 2011-07-16 at 19:27. |
quote |
Posting Rules | Navigation |
|
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
The Great AppleNova Server Move of 2008 | Brad | Feedback | 50 | 2008-04-14 18:33 |
how to move iTunes music files to ext hd, KEEP 'dir. structure' on pb? | malcolm | Apple Products | 13 | 2007-01-11 12:08 |
Move files automatically from old Mac to new Mac? | Squadra T | Apple Products | 2 | 2006-04-12 12:01 |
Enabling GD Image on a Linux server. | Jerman | Genius Bar | 3 | 2005-08-03 04:26 |
should i move my files? | bastard | Genius Bar | 8 | 2005-03-03 22:40 |