The future of data storage is still magnetic tape (2018)

> The media itself is cheap and durable, but the drives are $1,000 or more and sometimes break--in other words, if you really must have your data, you want two drives.

Newer drives will also drop support for older LTO standards, so you end up having to rely on flimsy old hardware to recover your older tapes. For most home users, it just doesn't seem worth the hassle. Maybe it starts making sense for casual/non-professional users in the 100TB+ range, but even the most committed data hoarders are nowhere near that.

Depends on the needs. Tape will keep longer unpowered. A traditional HDD is a bit riskier if left offline and unpowered for a while. Even if they're all powered in a tower, I'd still recommend something like Backblaze for redundancy.

in other words, if you really must have your data, you want two drives.

On the bright side, since it's removable media, anyone with a working drive can read your tapes for you.

You can restore the state of the drive to an arbitrary time in the past by replaying past writes.

For example, suppose something bad happens (a bug deletes important files, someone installs ransomware, etc) and you want to roll back the system three days.

Restore the last full backup you have that is older than the desired rollback time. Then replay from the write log every write that occurred between the time of that backup and the time you are trying to rollback to.

Another use for it is restoring or reverting files that were originally created more recently than your last regular (full or incremental) backup.

Example: I do incremental cloud backups once a week, on Saturday. Suppose I create a file on Monday, make various changes to it on Tuesday and Wednesday, accidentally delete it or corrupt it on Thursday, and realize this on Friday and want to recover the file.

With the "every write" log, I could find the file in the write log and recover it.

You might at first think that I would not be able to find it because unlike most backup programs the write log does not explicitly record file information. But it doesn't have to explicitly record file information because when you create a file or modify it the OS stores that information by writing to the disk, and those writes end up in your tape write log.

If I'm trying to restore /home/tzs/important_file to the state it was two days ago, the restore process would scan the write log starting far enough back to find the latest earlier write of the root inode (let's assume it was a classic Unix filesystem). Once it has that it can find /home's inode, then can find the correct contents of that on the target date, and so on until it has the inode of important_file. Then it can find what blocks comprised the file, and pull the right versions of those blocks from the write log.

Note that for this to work, the restore program must intimately understand the filesystem whose write log it is working with. This is one of the reasons you want to still do a more normal full/incremental period backup system with a backup program that explicitly stores files. The write log is best used to supplement the traditional approach.

PS: there was a backup program in whatever AT&T Unix was current in the early '80s (System III) that used the write log approach, although it was not continuous and it wrote redundant data. It was used for incremental backups. You started with a full block level backup. Then when you wanted to do an incremental backup it did this:

1. Make an empty list of blocks.

2. Add the block number of the superblock to the list.

3. Scan all the inodes looking for any inode that has a create or modify time later than the previous backup. For any such inode:

3a. Add the number of the block that contains that inode to the list.

3b. Add the block numbers of the file that the inode refers to to the list, including any indirect blocks.

4. Sort the block number list.

5. Copy all the blocks that are referenced in the list to the tape.

(There was probably a "write some metadata about the backup to the tape" step in there somewhere, too).

Note that this approach was redundant, in that if you changed part of a file the whole file would be written to the incremental backup. A pure write log would only write the changed blocks, but most systems don't record sufficiently granular modification information to allow that.

On the plus side, it meant that restoring a file from an incremental backup was easy. The file was either in that incremental entirely or not at all.

Kind of like asking the cost of a car + groceries.

You can pick up an old tape drive, like an LTO5, for about $1,000. You can LTO-5 tapes for about $15 each, with a capacity of 1.5 TB.

This setup puts the cost at around $10/TB, compared to somewhere around $30/TB for disk. Since you save $20/TB by using tape, it takes you 50TB before the total setup is less expensive than disk. Plug in your own numbers if you like.

You'll want to get a fiber channel card too, but that's easy enough.

Newer generations of LTO have marginally cheaper media (per TB) when you're buying in small quantities, but the drives are more expensive, and it's not uncommon to see poor capacity utilization in tape systems. The big players with lots of data will get newer LTO generations because they can save on library space, vault space, and drive time.

The main issue is that the standard was written back when file formats and tape formats were essentially the same thing. Data was read from tape, processed and written to a second tape. The memory of machine at the time were of the order of 1 block.

Decoupling tape formats from file format effectively resolves the issue, that is: read the tape in 6k blocks, write the file to a modern file system. Set the tape block size to 10mb. Write the file back to tape as a tar. You get the bandwidth limit of the tape drive.

The issue is almost all legacy industry software is still designed to read and write directly to tape objects so your tape isn't readable in all the software this data is designed for.

I am talking about bottlenecks on reading from the tape. Sending the data to /dev/null eliminates any write bottlenecks and we still max out at 60mb/sec on a 250mb/sec drive.

OK, cool. My browser's blind to MS, but I found an alternative: https://techxplore.com/news/2020-09-microsoft-holographic-so...

Magnetic tape is typically used for backup or archival purposes most of the time though, so that might not be such a big issue and they'll be used for a similar purpose

That is a feature, not a bug, for long-term archival media.

I regularly bring up the subject of physical write-enable switches (that hard drives used to have). Inevitably, someone responds that it's a great idea, and their software has a write-enable setting.

The other mind-boggling thing is I argue that remote updating shouldn't be allowed unless there's a physical switch to enable it. The response is that remote updating is necessary to keep systems secure from malicious remote updates.

I have a paper sleeve that goes over my webcam when not in use.

Yes, you said read write, but to answer your question at an industrial level, set your AWS IAM permissions to write only for your normal backup role. No delete or modify.

Not "simply" when the densities are so different.

They both happen to be similar in price, but if the density gap grew by 3x then suddenly tape would be more expensive and only used for certain types of archival.

From personal experience you are not going to be able to saturate even a horrifically obsolete LTO drive and SAS interface without specialized software or purely sequential data. Most of the cheap/free ways of running tape drives aren't optimized for parallel I/O and will horrifically shoe-shine (that's the rev-up and rev-down sound you hear when the tape drive isn't getting enough data), which isn't great for the tapes and massively increases backup time and drive usage.

AFAIK most commercial tape deployments nowadays are disk-to-disk-to-tape arrangements. All the actual data is serialized to an archive on disk first, and then that serialized archive is written to tape at full speed. This minimizes tape wear and ensures your very expensive tape drives are being used efficiently.

Very insightful comment on the data rate, if you can't feed the LTO properly, it may be stopping and starting all the time, which could have a detrimental effect to its mechanism. But this is probably easily fixed by spooling (using bacula terminology). You buffer like 100Gbytes, write them in one batch. Plus, some drives support variable speeds.

Also, in addition to SAS, a fibre channel card would also fit the bill, albeit probably a trifle more expensive. But if you go for low speeds (e.g. 8Gbps), those cards can be less than 100$.

> Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer.

Maybe. Keeping up only really matters if your slowest data rate is going to be between 50MB/s and 120MB/s. If it's below that then you have a strong need for a buffer drive no matter what tape generation. And if it's above that then you shouldn't have any real trouble no matter what tape generation.