User Name
Password
AppleNova Forums » Genius Bar »

Best way to move files from Mac to Linux server?


Register Members List Calendar Search FAQ Posting Guidelines
Best way to move files from Mac to Linux server?
Thread Tools
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2011-07-06, 20:55

So what is the best way to move something like 314351, files all small but totaling about 20GB? FTP just isn't cutting it. Would rsync be faster or just different? This is from my home system to my dedicated web server.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2011-07-06, 21:00

Can you pack the files together and explode them on the destination?

I'm thinking something along these lines would work:

On your Mac:

cd /path/to/parent/directory
tar -czvf archive.tgz filesdirectory
rsync -v archive.tgz remoteuser@remotehost:/path/to/destination/parent/directory

On the Linux server:

cd /path/to/destination/parent/directory
tar -xzvf archive.tgz

If you wanted to do it all from your Mac, you could even send the "tar -xzvf" command remotely over ssh.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2011-07-06, 21:04

Also, you might want to give the rsync command the arguments "--partial" and "--progress". The first allows you to quickly resume if the transfer gets cut prematurely, and the second gives percentage progress output.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2011-07-06, 21:09

BTW, I suggest packing the files together because that will be much faster than sending them separately. Less handshaking around file starting and ending bits and less IO access as the reads and write skip around the drive.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2011-07-06, 21:24

I had a feeling this was the answer, thanks Brad.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2011-07-06, 22:16

Working perfectly Brad. Looks like about 6 hours for the upload! I can only imagine what it would have been with FTP and individually.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2011-07-08, 20:05

Due to server HDD size it looks like I'm going to have to do individual files. It's 16GB as a tarball and 19GB on site expanded. With my other sites on there I just don't have room to copy over from within the server. I'm going to try to set up a sync option instead. Maybe it'll work well this way.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
Gargoyle
http://ga.rgoyle.com
 
Join Date: May 2004
Location: In your dock hiding behind your finder icon!
 
2011-07-10, 03:01

Why can't you just select all the files in Transmit or something and hit upload (then go for a coffee... or a holiday!) ?
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2011-07-10, 14:08

Quote:
Originally Posted by Gargoyle View Post
Why can't you just select all the files in Transmit or something and hit upload (then go for a coffee... or a holiday!) ?
Well, you could, but that's not as easily automatable and not nearly as efficient.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2011-07-10, 20:00

That and a holiday is needed for these files and the time it takes. I've gotten rsync to work well for this with the help of Brad. It's going to make my life a whole lot easier.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
Gargoyle
http://ga.rgoyle.com
 
Join Date: May 2004
Location: In your dock hiding behind your finder icon!
 
2011-07-11, 07:14

I didn't get the feeling that this was needed to be repeatable, more of just a one-off.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2011-07-11, 08:12

Well, I wasn't sure when I first posted this. I use Transmit which I can do Automator actions for, but rsync is definitely the best option at this point.

Thanks for your help and input anyway.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2011-07-16, 18:57

So I've written the sudo code for this, now to actually code it.
Code:
MC Maps Pseudo code Run backup script on Lenny Move files backed up from Lenny to Beast <- Local network Expand files into World RAM folder. Run map making jar Rsync results to mcmaps.applenova.com <- Remote network Replace index.html and specs.js with AN versions <- Remote network
How can I do a script that would allow me to automate the remote connection without setting a user to no password? I could create another user and have it only set for accessing that folder, but I'd rather have the same user I'm using now.

What options do I have for the cross computer connections? Moving the backup to Beast can be an FTP thing I think, but would rather rsync. So yeah, I'd like rsync for anything that's changing machines, but how do I do it without a password or can I program a password into a script? Sure it's not that secure to have a password in plain text on a script, but these machines aren't web accessible in the locations for the password.

Edit: Searching I found this, any thoughts on it? I figure I'll give it a shot.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.

Last edited by turtle : 2011-07-16 at 19:27.
  quote
Posting Rules Navigation
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Post Reply

Forum Jump
Thread Tools
Similar Threads
Thread Thread Starter Forum Replies Last Post
The Great AppleNova Server Move of 2008 Brad Feedback 50 2008-04-14 18:33
how to move iTunes music files to ext hd, KEEP 'dir. structure' on pb? malcolm Apple Products 13 2007-01-11 12:08
Move files automatically from old Mac to new Mac? Squadra T Apple Products 2 2006-04-12 12:01
Enabling GD Image on a Linux server. Jerman Genius Bar 3 2005-08-03 04:26
should i move my files? bastard Genius Bar 8 2005-03-03 22:40


« Previous Thread | Next Thread »

All times are GMT -5. The time now is 06:43.


Powered by vBulletin®
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004 - 2024, AppleNova