User Name
Password
AppleNova Forums » Purchasing Advice »

Can external hard drive reduce main drive's burden?


Register Members List Calendar Search FAQ Posting Guidelines
Can external hard drive reduce main drive's burden?
Thread Tools
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-04, 08:08

I'm trying to figure out what/whether it is possible to help me be more productive by parceling out the HD-intensive work so I can continue to do my other stuff. My workflow lately has me running several queries, while I want to continue on development as I wait for the results. As things are right now, I get beachballs a lot if I try to multi-task and I do worry that I'm hastening along my drive's eventual demise.

But I'm not sure if an external hard drive would solve the problem? From what I can tell in my activity monitor, I have processing power to spare (generally 50-60% idling, though it seems to flip-flop between the cores) while my hard drive is quite busy, but I do worry about the extra overhead and lower throughput being a factor that may make the addition pointless. Should I be looking at say, a mini, if I really wanted to parcel out the HD tasks? Or a certain kind of enclosure (which obviously won't be USB; most likely either FW or gigabit ethernet)?
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2009-03-04, 10:06

It would help to know just a little more about your specs.

For my MacBook Pro, I bought an external hard drive (specifically, a Western Digital MyBook Studio) a while ago, and an eSATA ExpressCard to go with it. This makes quite a bit of a difference when running VMs, which I install on the external drive.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-04, 10:32

I really, really wish I had a SATA port.

It's a 24" iMac, the first generation of that aluminum/black iMac, 4 GB of RAM, 230 GB 7200RPM HDD with 2-something ghz intel C2D. I do have a FW800 or FW400 available.

What I'm really looking for is a reassurance that external hard drive is all I need for offloading the HD works within the VM to external hard drive without slowing down the everything else (though I do expect there to be slightly more overhead and more back-forth between the processor and the hard drive... just not sure how much and whether it would justify the effort)
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 10:21

Bump.

A reassurance or some more information/options would be great.
  quote
bassplayinMacFiend
Banging the Bottom End
 
Join Date: Jun 2004
 
2009-03-05, 10:45

Why not put whatever you're querying on an external drive and benchmark the results? A simple stopwatch should be efficient enough for benchmarking before/after scenarios.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 10:54

Great idea!

Except I've not actually bought anything and therefore have nothing to test.

I suppose I could go to BB and test it, and return if I didn't like the results; the problem is more that I don't exactly have top two or something narrowed to make the self-comparison worthwhile, not to mention feeling uncertain that this is even the right product for my needs.
  quote
bassplayinMacFiend
Banging the Bottom End
 
Join Date: Jun 2004
 
2009-03-05, 11:10

I have at least 3 external drives floating around my house so I wasn't thinking you may not have any you could use at least temporarily to see if your idea is worth investigating. My bad.
  quote
Taskiss
Veteran Member
 
Join Date: Jul 2007
Location: St. Louis, MO
 
2009-03-05, 11:21

Quote:
Originally Posted by Banana View Post
I do worry that I'm hastening along my drive's eventual demise.
Personally, I wouldn't worry about using the main drive. I've never seen the life of a drive affected by utilization as far as I could tell, there have been drives fail on my busiest volumes and on my least, with no real correlation between utilization and failure rates.

I'd use and abuse the main drive without mercy, make sure there's a time machine backup of the main, and off-load unused files to an external drive.

real hackers don't use sigs
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 11:23

Actually, it's not about the lifetime of hard drive but rather improving my workflow by reducing the waits for I/Os between various tasks. As things are right now, I beachball too often when I have HD-intensive tasks running in background when I want to do something else.
  quote
ezkcdude
Veteran Member
 
Join Date: Jan 2005
 
2009-03-05, 11:44

I think the Gigabit ethernet may help. In my old lab, we had a XRAID with the fiber connection, and it was actually faster to download files from the server than to access the same file from your own hard drive. It's not exactly the same situation, but you get the idea.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 11:49

Cool, thanks for the confirmation.

I was kind of wondering about the bottleneck. SATA drives are supposed to have 3 Gb/s, meaning that it'd outrun any connection, be it gigabit ethernet or FW800, but on my own hard drive, the activity monitor is telling me otherwise. It peaked at only 20 MB/s, which would be easily handled by either connection. Thus, my confusion.
  quote
PB PM
Sneaky Punk
 
Join Date: Oct 2005
Location: Vancouver, BC
Send a message via Skype™ to PB PM 
2009-03-05, 12:36

Have you checked RAM usage during intensive tasks? It honestly sounds like you are hitting the RAM sealing, not a hard drive issue.

The theoretical peak throughput of SATA II is 3GB per second, but even top of the line hard drive peak at 150MB/s, with average ones in the 60-90MB/s range. That does seem slow, but it does depend on the size of the file, and how much read and write activity is going on. Also, a 230GB (Do you mean 320GB, I've never heard of a 230GB drive), isn't going to be super fast. If there is intense read and write you could easily see slow access times (20MB/s), especially if the drive is offering up a lot of virtual memory.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 12:48

Sorry, that was my typo, you're right- it's 320 GB.

Well, looking at my RAM (I have 4 GB of it), I have 61 GB of VM, <2.5 GB of paging both ways with 1.7 GB being used.

I'm not quite sure whether it means that I'm hitting the ceiling...
  quote
PB PM
Sneaky Punk
 
Join Date: Oct 2005
Location: Vancouver, BC
Send a message via Skype™ to PB PM 
2009-03-05, 13:41

Yeah, and I cannot use the right ceiling...

2.5GB of paging space isn't bad, but it does mean there isn't enough RAM and that the OS is using the hard drive for additional memory, which would slow you down big time. I also have that problem when I am working in Aperture, and sadly short of getting a new Mac loaded with 8GB of RAM I know I wont be getting away from it. Having your files on another drive will help, but I'm not sure it will make too much difference.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 16:24

Two things that needs to be considered:

1) I usually run a Virtual Machine all time, consuming 1.5 GB, leaving 2.5 for Mac OS X.

2) Within that VM, I usually run SQL Server, which is where I do bulk of my work, which can include querying upon millions of rows.

Generally, if it's just VM running with few programs, my system is responsive enough for me. It's when the SQL Server is running queries, which I assume are HDD intensive, does it start beachballing a bit too much for me. Not to mention that I'd be rather keep working away while it works in background, but it's kind of impractical as it is now with SS running in background...
  quote
PB PM
Sneaky Punk
 
Join Date: Oct 2005
Location: Vancouver, BC
Send a message via Skype™ to PB PM 
2009-03-05, 17:05

As I said, having the VM on an external drive would help, I'm just not sure it would be that much. A USB or FW drive will have less throughput, and you might even experience more lag, at least for the VM. I cannot say for sure though since I've never tried doing that with a VM.
  quote
Taskiss
Veteran Member
 
Join Date: Jul 2007
Location: St. Louis, MO
 
2009-03-05, 18:40

Quote:
Originally Posted by Banana View Post
Two things that needs to be considered:

1) I usually run a Virtual Machine all time, consuming 1.5 GB, leaving 2.5 for Mac OS X.

2) Within that VM, I usually run SQL Server, which is where I do bulk of my work, which can include querying upon millions of rows.

Generally, if it's just VM running with few programs, my system is responsive enough for me. It's when the SQL Server is running queries, which I assume are HDD intensive, does it start beachballing a bit too much for me. Not to mention that I'd be rather keep working away while it works in background, but it's kind of impractical as it is now with SS running in background...
http://www.mactech.com/articles/mact...rks/index.html
Quote:
Many people have the feeling of "more is better," but clearly when it comes to RAM in the virtual machine, that is not necessarily the case. More RAM means longer virtual machine launch times, suspends and resumes. For most users, 1GB of virtual machine RAM will work best. Use more than that only if you really know you need it.
How many CPU's have you given the VM? Also, if I were you, I'd either run the SQL server on another box or I'd run it in bootcamp. You have some percentage of overhead running a VM, so that's where I'd look to improve things. Since the bulk of your work is using SQL server, well, then cut out the overhead of the Mac OS.

real hackers don't use sigs
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 20:03

That was a great and informative article, even though it ended up making me feeling bad that I chose Fusion over Parallels.

The VM only has one virtual processor.

You may be correct about using Boot Camp, but there is a reason for virtualization instead; some of my workflow move over to Mac OS X and I find myself more productive when I make use of various OS X applications not to mention the fact that my VM doesn't have any "fearware" running, saving on resources because all network traffic has to go through OS X.

I pointed earlier that my processor were idling 50% of the time even when the VM/SQL Server was busy, so I'd expect it to not beachball so often, but it does, presumably due to two OSes fighting over I/O on the same drive (?).
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2009-03-05, 20:18

Quote:
Originally Posted by Banana View Post
I pointed earlier that my processor were idling 50% of the time even when the VM/SQL Server was busy, so I'd expect it to not beachball so often, but it does, presumably due to two OSes fighting over I/O on the same drive (?).
As I mentioned in the other Safari thread (that I assume prompted you to post this), that's exactly my suspicion.

If you're waiting on SQL to churn through tables with millions of rows, unless you have greatly tuned your database settings, it will probably only read small parts into memory and thrash your disk like crazy. That will lead to IO waits which may cause operations on the Mac side to hang until they can gain access to the disk.

How big is your database? (In bytes, not rows.) If you have tons of free memory in your VM, you could try copying your database files to tmpfs (assuming you're running a Linux VM), tell your database to look there for its files, and retry. tmpfs is basically a RAM disk. If suddenly everything flies once you've copied the files into that, you've got your answer.

Of course, you don't want to use tmpfs long-term because it's volatile memory. If something crashes or your VM halts, you lose everything in there. That's why you should copy the files there, not move them.

Also, you're indexing the columns that you're primarily querying against, right?

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 20:26

The test database I'm developing against is >20 GB. Not quite big, as it's just sample data I use for testing the prototypes that eventually becomes stored procedures for the current project I'm working on.

Believe me, I'm sorely missing MySQL's fabulous memory engine as SQL Server has no similar thing (yes, it does have temporary table but they get written to HD anyway. How assine is that?)

As for indexing, I make it a point to do full table scan for everything I do. It's much more fun. You should try it sometimes.
Yes, though there are times where I'm adjusting the indexes, figuring out if it helps or not but that's expected as part of development.
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2009-03-05, 20:32

Sweet jesus, man. 20 GB? What on earth are you keeping in there?

No wonder your drive is getting hammered!
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-05, 21:25

That's just sample data. The actual data is supposed to be much more than that... It's primarily data for business reporting for a corporation I'm contractor for.

So farming out VM to an external drive should be good enough for relieving my Mac of beachballs, then?
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2009-03-05, 23:25

Niiice. I'm used to working with a ~30 GB database at my day job; the biggest table in it has about 15 million rows.

Anyhow, I'm sure moving that stuff off to an external drive would help you in some measure. Will it be a cure-all for your spinning beachballs? I can't say, but the general idea is sound and it's the best potential solution, I believe, given your description of the problem, short of moving the whole thing onto a dedicated machine.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2009-03-06, 00:28

Quote:
Originally Posted by Banana View Post
Believe me, I'm sorely missing MySQL's fabulous memory engine as SQL Server has no similar thing (yes, it does have temporary table but they get written to HD anyway. How assine is that?)
#-prefixed tables do (and I don't find it asinine; we have a need for temporary but disk-backed tables), but have you tried @-prefixed ones, as created with DECLARE TABLE @foo? I'm not sure right now, but I don't think those end up in tempdb.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-06, 01:20

I'm aware of table variables; unfortunately, they do not fit my requirement. Otherwise, I'd be more than happy to use them. My offhand comment was rooted in frustration over the limitation of table variables and less control over how data are cached compared to MySQL.

But that's another topic!

Thanks, BTW, for sharing your input with your external drive.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2009-03-06, 02:51

I suppose you could always trick SQL Server by putting tempdb on a RAM drive?

*looks around*

In fact, that idea seems less far-fetched than I had thought: http://www.google.com/search?q=ram+drive+tempdb — I skimmed over some of those hits and they all seem like worthwhile reads.

Quote:
Originally Posted by Banana View Post
Thanks, BTW, for sharing your input with your external drive.
Well, my situation is somewhat similar. The main applications I use are Visual Studio and SQL Server Management Studio. The locally installed SQL Server is only for testing, so is only sample data on it, but I nonetheless easily run into I/O issues — it is clearly by far the greatest bottleneck.
  quote
Taskiss
Veteran Member
 
Join Date: Jul 2007
Location: St. Louis, MO
 
2009-03-06, 06:24

I'm wondering what kind of performance hit you will take, moving that DB off to an external disk. You may not see as many I/O contention issues, but the bandwidth might suck.

real hackers don't use sigs
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-06, 07:39

Quote:
Originally Posted by Taskiss View Post
I'm wondering what kind of performance hit you will take, moving that DB off to an external disk. You may not see as many I/O contention issues, but the bandwidth might suck.
Yeah, that's one of my concern.

Quote:
Originally Posted by Banana View Post
I was kind of wondering about the bottleneck. SATA drives are supposed to have 3 Gb/s, meaning that it'd outrun any connection, be it gigabit ethernet or FW800, but on my own hard drive, the activity monitor is telling me otherwise. It peaked at only 20 MB/s, which would be easily handled by either connection. Thus, my confusion.
Quote:
Originally Posted by PB PM View Post
The theoretical peak throughput of SATA II is 3GB per second, but even top of the line hard drive peak at 150MB/s, with average ones in the 60-90MB/s range. That does seem slow, but it does depend on the size of the file, and how much read and write activity is going on. Also, a 230GB (Do you mean 320GB, I've never heard of a 230GB drive), isn't going to be super fast. If there is intense read and write you could easily see slow access times (20MB/s), especially if the drive is offering up a lot of virtual memory.
Assuming PB PM's assertion is correct, if I bought the best drive, it would saturate the bandwidth, but not by that big margin as I had though (150 megabytes/second = 1.2 gigabits/second), and that's just a peak speed, which is not attained all time.

I'd like to see if anyone has seen bandwidth become the new bottleneck in such case, though.
  quote
Banana
is the next Chiquita
 
Join Date: Feb 2005
 
2009-03-06, 07:42

Quote:
Originally Posted by chucker View Post
In fact, that idea seems less far-fetched than I had thought: http://www.google.com/search?q=ram+drive+tempdb — I skimmed over some of those hits and they all seem like worthwhile reads.
Cool. I had thought about that for another application, but last time I looked around, RAM drives seemed to be out of vogue in Windows world and got cold foots. Will definitely re-consider as in my development, there's really no need for disk-backed temporary tables.
  quote
Taskiss
Veteran Member
 
Join Date: Jul 2007
Location: St. Louis, MO
 
2009-03-06, 08:12

I might try keeping the tempdb on the main drive if you move the db off to an external - depending on how your queries are written, you might want that on the fastest drive available...RAM would be best if you have the space though. I'd also be concerned with the sheer number of I/O requests saturating the external disk controller more than the bandwidth. Another reason to consider leaving the tempdb off the external drive.

real hackers don't use sigs
  quote
Posting Rules Navigation
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Post Reply

Forum Jump
Thread Tools
Similar Threads
Thread Thread Starter Forum Replies Last Post
External hard Drive Mac Lover Genius Bar 10 2008-01-20 23:54
External Hard Drive ? m4hl3r Third-Party Products 19 2008-01-08 16:15
External Hard Drive bmdbaseball Purchasing Advice 6 2007-05-26 13:44
OWC mercury elite pro external multi drive -hard drive- case rcs Purchasing Advice 2 2005-09-18 18:15


« Previous Thread | Next Thread »

All times are GMT -5. The time now is 18:22.


Powered by vBulletin®
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004 - 2024, AppleNova