[Palomine]

Yes, I should google at computer forums for info on this, but BEA users often know even the most esoteric stuff and if they say something's true, I tend to trust 'em since we've got history.

What's the deal with losing so much of a hard drive's advertised capacity once it's formatted? Like 10% (sometimes more, sometimes less) of the storage space is essentially unavailable for use once you format it. My 3.0TB drive has just over 2.8TB usable (once formatted) and a 5TB external USB drive I was reading about at Newegg shows up as only 4.54TB when it's being used in Windows. That's a lot of space going to waste, no?

I read something somewhere about how some OSes use binary vs decimal addressing being responsible for this, but it wasn't clear. Other people suggest there are ways to avoid this: to be able to use more of the space on the drive. I remain unclear about it. I'm not an expert on futzing with partitions and such, but am willing to learn if necessary.

All the drives on my PC are formatted using the "ext4" standard, which is the default/most common one for Linux (at least for Debian-based distros)... however, I can of course format drives as FAT32 and other standards (though IIRC, there are some significant filename length/character restrictions when using FAT32, whereas ext4 seems to accept most ANY absurd/ancient filename w/o complaint, even those with odd or null characters in them (say, from an old archive)) so I usually use ext4. Also, IIRC when I have formatted drives as FAT32, I've noticed a similar loss of available space on a drive.

The system drive on my PC is an SSD (128G advertised size but only about 108G available to me). All the rest of the drives are regular hard drives: a mix of internal SATA and external USB drives (generally SATA inside) and all seem to "lose" between 8 and 15% of their storage post-formatting.

I'd appreciate any info/advice about this... obviously, I'd like to reduce the amount of wasted drive space for future drives (or for existing drives, even if it meant reformatting them and then restoring all their data). Thanks in advance for any pointers/links/summaries about this you can offer.

Edited later to show a screencap of disk properties for a 5TB WD external USB 3.0 drive for which Properties shows the full 5TB of space is available for use (that 18.3GB used is a copy-in-progress to the drive). This drive came formatted NTFS from the factory, which is how it remains at present. I can't help but suspect that were I to reformat it Ext4 (the standard filesystem for modern Linux PCs) and then check Properties again, I'd have "lost" some percentage of that available space. Further, I'm not even sure that if I reformatted it later back to NTFS that I'd "regain" the "lost" space... somehow, I wonder if the factory format is different from what I can do, even though that makes no sense at all.   

[SwitcherX]

HD makers define a megabyte as 1,000,000 bytes, not 1,048,576 bytes.  Those ~48k add up fast.

[BacE]

1Gigbyte=0.931322575 Giga byte
1Terbyte=0.909494702 Tera byte
TB=(10^3 / 2^10)^4
GB=(10^3 / 2^10)^3
So the difference becomes greater with each greater unit
ie PB=(10^3 / 2^10)^5=0.88817842 Pet-a byte (damn auto censor)

sorry for the dutch wiki link, but table is more extensive than the english
https://nl.wikipedia.org/wiki/Kilobyte
B"I've been explaining this for 20years  "acE

[TheZookie007]

HD makers define a megabyte as 1,000,000 bytes, not 1,048,576 bytes.  Those ~48k add up fast.

Yeah, basically it's the difference between using base 10 and base 2 to count stuff.

A gigabyte (GB) = 1000 x 1000 x 1000 bytes = (10³)³ bytes = 10⁹  bytes = 1,000,000,000 bytes.

A gibibyte (GiB) = 1024 x 1024 x 1024 bytes = (2¹⁰)³ bytes =  2³⁰ bytes = 1,073,741,824 bytes.

Since hard disk manufacturers measure capacity in GB instead of in GiB (a "giga-binary-byte", as it were), they are basically rounding down from the real capacity of the drive.

[Hiram]

Yeah, basically it's the difference between using base 10 and base 2 to count stuff.

I've been explaining this to people for sometime, and that is the most succinct sentence. As long as they understand binary.   

[Palomine]

Is that the ONLY reason? And if so, why does the percentage of lost space vary on different sized drives?

Are you guys saying the different counting systems account for the difference... i.e.: the manufacturer counts one way and the (makers of) operating system another?

Nothing to be done to reduce the percentage lost? I knew about the numbering thing, but was under the (mistaken?) impression that there were other factors involved in the loss of usable space. So, you're saying that there's NO way a user with a given OS (OSX, Windows, Linux) can reduce the amount of lost space? I just ordered some new drives and if there's some way to reduce the lost space, now's probably the time to do so (before I fill them with data).

Thanks for helping my non-computer-scientist brain grok this!

[Hiram]

Are you guys saying the different counting systems account for the difference... i.e.: the manufacturer counts one way and the (makers of) operating system another?

Yes, that is my understanding. Any other variations could be due to different disk formats and data block sizes.

http://www.cnet.com/news/available-hard-drive-space-block-sizes-and-size-terminology/

[TheZookie007]

Is that the ONLY reason? And if so, why does the percentage of lost space vary on different sized drives?

Are you guys saying the different counting systems account for the difference... i.e.: the manufacturer counts one way and the (makers of) operating system another?

The binary/decimal thing is one part of the reason. Another part of the reason is (as Hiram said) the differing block sizes and disk formats. Another part of the reason is that part of the drive is reserved for diagnostic data (such as is used by S.M.A.R.T.), and indexing data.

[Palomine]

Alright, I admit that I still don't think that I understand the real deal about this: the varying discrepancy between advertised drive capacity and actual/usable drive capacity after formatting with filesystem X vs filesytem Y, etc... You'd think that by now it'd be possible to state some "rules" ...at least for the most common scenarios, such as "when formatting with NTFS, you'll lose 7.6% of advertised capacity" or somesuch.

My ignorance notwithstanding, those WD 5TB drives I bought recently arrived NTFS and that's how I left them: NTFS. I did *not* reformat them at all but just used them as-is out of the box. And according to my OS (Ubuntu) the available room on the drive is equal to the manufacturer's advertised capacity (5TB) which, IIRC, is something that I've not seen before (or at least not often).

I do NOT know if the usable space would have decreased had I reformatted the drive NTFS after I got it (no reason it should have, but somehow, I wouldn't have been surprised) nor do I know what the available space situation would be had I reformatted with a different filesystem (like Linux's Ext4 for example). I was curious, but I played it safe and just left the drives alone, since I wanted every byte of room.

Maybe WD makes the drive a little bigger to offset formatting losses, so the customer who buys a 5TB drive actually has all 5TB to use? I dunno.

[SwitcherX]

Another thing to keep in mind is that when you format a drive with a file system, the file system itself takes up space.

If a drive is advertised as 4GB:

1) HD makers use 1GB=1000^3 bytes, not 1024^3
2) 4GB is when it's unformatted

[Palomine]

Another thing to keep in mind is that when you format a drive with a file system, the file system itself takes up space.

If a drive is advertised as 4GB:

1) HD makers use 1GB=1000^3 bytes, not 1024^3
2) 4GB is when it's unformatted

Except in the case of the WD 5TBs I just purchased apparently, which show as 5TB available when formatted. It's the inconsistency/unpredictability that bothers me.

[Robert]

Another issue is that ext4 has a concept of reserved space.
Reserved space is unusable by anyone but root.

This is partially to help with file fragmentation and to ensure root has disk space in case something goes wrong.
It's important on the root partition so root can always log in, but the 5% number was picked when disks were much smaller.

On my disk:
sudo tune2fs -l /dev/sdc1 | grep "Reserved block count"
Reserved block count:     12209510

I have the reserved percent set to 0 on my non-root partitions and about 1% on my root. 
The second answer is better than the first.
https://superuser.com/questions/444269/display-filesystems-free-space-available-to-the-root-user

[Palomine]

Another issue is that ext4 has a concept of reserved space.
Reserved space is unusable by anyone but root.

This is partially to help with file fragmentation and to ensure root has disk space in case something goes wrong.
It's important on the root partition so root can always log in, but the 5% number was picked when disks were much smaller.

On my disk:
sudo tune2fs -l /dev/sdc1 | grep "Reserved block count"
Reserved block count:     12209510

I have the reserved percent set to 0 on my non-root partitions and about 1% on my root. 
The second answer is better than the first.
https://superuser.com/questions/444269/display-filesystems-free-space-available-to-the-root-user

Thank you very kindly for the information Robert!

As mentioned, I no longer user Ext4 for most of my drives... they're mostly running the NTFS that they came with out of the box. Only my system drive (a relatively small SSD) is formatted Ext4 since that's what Ubuntu Linux seems to use by default, and losing some usable space on the system drive (which I never let get filled up of course) isn't as big a deal as losing space on the bigger data drives, where you want to squeeze in every last file possible.

I remember some years back when I switched to Linux (after more than 20 years mainly using MacOS) that I did encounter and look into the business of Linux setting aside that 5% of space, which exacerbates the "lost drive space" issue. I admit that being new to Linux, I was reluctant to mess around with sudo or deeper settings, so I never did anything about it. Also, I was never clear (and still remain unclear) whether that 5% reserved space could be reduced to 1% (or whatever) once the drive had already been formatted and was in use, or instead if it had to be done in conjunction with wiping and re-formatting the drive.

Out of curiosity, I did cut/paste your "sudo tune2fs -l /dev/sdc1 | grep "Reserved block count"" command into my terminal window, changing your "sdc1" to "sda1" which is the SSD boot drive on my 10-year-old PC:

sudo tune2fs -l /dev/sda1 | grep "Reserved block count"
Reserved block count:     1413670

Since I don't recall actually doing anything to change the 5% figure when I set up this SSD boot drive a few years ago, I assume it's still set at 5%, or whatever is the default in Ubuntu 14.04LTS (which is what I'm running... I just use the long term support versions, not the in-between versions). On my system, 1.4 million blocks are reserved vs. the 12 million blocks on your system, but I assume that's due to the fact that you're probably using a much larger system drive than I am, so a lower percentage set aside as reserved would still mean a bigger reserve than on my machine. At least, that's my layperson's imagined explanation.

On the off-chance it tells you anything useful, attached is a screencrop from the Disks utility, showing info for my SSD boot drive (120GB Kingston SSD, Master Boot Record partitioning). Thanks again for your post and for anything else you'd care to share.

[Keisha_Evans]

Loved all the info!  Everything I could think of to mention was already covered, lol.  Bits vs bytes, restricting space, etc.  Fun stuff!  I'm only now messing around with Linux.  Hope you got it all worked out!