View Single Post
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2022-08-06, 19:13

Quote:
Originally Posted by drewprops View Post
I'm copying an external 4TB drive to a different external drive on my 2021 M1 MBP and still have 200,000 files left to copy after 23 hours.

Would it take this long on an NAS?

EDIT:
Nevermind, I read up on the app I'm using and it doesn't just copy, it checks the files and such and long transfer times like this are typical, so I'm over here talking kumquats while you all are talking apples - my bad
...
Actually, it's still a good question, and I suspect that the underlying mechanism your app is using for copying isn't too different from mine. I use the command-line tool "rsync" to do most of my remote file copying ("remote" being either to locally connected external drives or across a network), and it also does some pretty thorough sanity-checking along the way, estimates transfer rates, shows progress, and even copies all the weird Mac-specific metadata, attributes, and fork stuff. In general, copying lots of little files is much slower than copying fewer large files. Unless I know that I'll frequently need to dig into certain folders with lots of little files (like an archived email inbox), I'll zip the folder into one big file, and the time to zip it and transfer the zip is still usually faster than transferring the original folder of files. This is because systems have to stop to record where each file starts and stops and write metadata like created, modified, and access times for each along the way, and although that's usually only a very brief delay, those tiny delays will definitely add up to noticeable effects when you're churning through hundreds of thousands of files.

But it might be even worse for you…

Unfortunately, many "smaller" (1-8 TB) drives now use a new kind of technology called Shingled Magnetic Recording or "SMR" which can make performance truly awful in some conditions, including when you're doing lot of writing. Many 2.5" drives and some 3.5" drives, both internal and external, from pretty much all drive manufacturers have been found using this technology. I wouldn't be surprised if you got stuck with one; drive manufacturers do not advertise which drives do or don't use this technology. It's so bad that IT communities and companies like Synology and iXsystems (the TrueNAS company) have documented warnings that users and customers should not use particular drives because they basically can't work in a NAS due to the performance issues. There was even a class-action lawsuit in 2020, but I don't know if or how that ended. (I'd bet the manufacturers won because they always do.)

For now, it seems that only the larger 10+ TB drives consistently use the older Conventional Magnetic Recording or "CMR" technology which is much more performant across the board.

edit: If you're curious to know more about CMR vs SMR, here's a quick summary from Linus Tech Tips: https://www.youtube.com/watch?v=aztTf2gI55k Jump to 6:15 if you want to see some "holy shit" performance differences.

Speaking of which…!

Earlier this week I dropped a fat wad of cash on two 14TB external drives for my newborn NAS. Whenever I get new a drive, the first thing I do is connect it to one of my desktop computers and run a full two-pass secure erase of the whole thing. After forcing every block of every sector to be written, I'm reasonably confident that I didn't buy a lemon. It is a painfully slow process for these giant drives, though, even over USB3. The first one took almost 24 hours, and I haven't started the second one yet. Once they've both verified good, I'm going to shuck them to install in the NAS.

And I do plan to share the shucking process here. It's pretty neat, and it's something I never really considered doing until the last couple years. I've had many external drives over the decades, but without a robust long-term storage plan in place, which I think I have now for the first time ever, going from one external to the next was perfectly fine for my needs.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote