Adding storage - Best options? (External USB drives, automatic decryption, media, etc.)
https://slrpnk.net/post/12360419
=> More informations about this toot | More toots from Guenther_Amanita@slrpnk.net
linuxserver.io/…/2017-06-24-the-perfect-media-ser…
I did perfect media server
It’s got mergerfs for splitting data and using disks in various sizes .and snapraid for a level of redundancy. Tho raid isn’t backup.
That said I’m now running this setup on a n100 machine with a qnap tl-800c jobs USB c box.
Works great for downloads / Plex and home server needs.
The b100 chip isn’t amazing… Don’t get me wrong but it works really well for Plex.
Hope this all makes sense. I’m on mobile with out my glasses. Lol
=> More informations about this toot | More toots from 7U5K3N@lemmy.dbzer0.com
Oh man, what a mess. It is just not worth it if you’re starting adding 1 or 2 TB. Also you don’t say what kind of data you want to store on this system. If it’s media files (static once written) that can simplify things.
I’d say don’t mess with external drives at all. Your simplest path is upgrade your 1TB internal SSD to 2TB or 4TB. Those aren’t too expensive, and you get SSD storage. Yes you may as well use LUKS unless you want to get fancier. I have some thoughts about key management but haven’t implemented them in practice, so talk about that would be theoretical.
RAID is for when you have data that changes, like databases where you frequently add rows or do updates, so you are up to date if a drive crashes just after an update. It also lets you keep the system running while you hot swap the crashed drive. If you don’t mind taking your storage offline while you restore from a backup, and you don’t mind having to recreate the most recent data, you don’t need RAID.
I simply keep my stuff on a Hetzner StorageBox, encrypted with Borg Backup. That eliminates all the hassles of RAID, buying hardware and keeping it at home, etc. I can remote mount it (read only) with sshfs with all cryptography happening on the client side (in practice I don’t do that very often). There’s no need to use an encrypted file system on the server, or for the server to ever see plaintext. Of course StorageBox is not self hosted, but you could do something similar with a bare iron storage server. Anyway I think it’s difficult to beat this for economy until you have tens or maybe 100’s of TB of data.
=> More informations about this toot | More toots from solrize@lemmy.world
+1 for borg + hetzner storage box, though externals do give pretty good value for some uses. I have all my movies/tv on a 6tb external and it would have cost so much more to do it any other way
=> More informations about this toot | More toots from Andrzej@lemmy.myserv.one
Buy external drives. Don’t run them in RAID, use one to store backups and plug it in once or twice a week to copy data to it.
The secret to RAID is that it doesn’t buy you data protection, it buys you uptime to access data while a device in the array is failed. This is most valuable to businesses that can’t afford the downtime that recovery from a backup incurs. The most paranoid RAID will still fail sooner or later, due to hardware or software failure, and as a home user with a limited budget you’re far better off having one offline backup that you can use to recover data from once that happens.
Backup only data you can’t afford to lose (eg: don’t backup downloaded data that can be replaced easily, like a game or movie collection) and your backups will be much more manageably sized and you won’t need to spend as much on your backup drive. If a backup disk is too much for your budget you can always exploit cloud backup plans, backblaze PC backup has no limit on the size of your backups and only charges something like ~$60/yr.
=> More informations about this toot | More toots from seaQueue@lemmy.world
This is pretty great advice to get into it. I previously ran 3 poweredge 2950s but have since switched to nothing self hosted and back to everything self hosted but on a much leaner setup with a NUC and 14tb WD my book drive with a dual Noctua 4020 fan shroud I 3d printed that it absolutely needed as I killed the original drive in two weeks.
My replica is just a 14tb in my desktop I run rsync to pull the data occasionally after checking SMART status on the primary. It’s not versioned or perfect but it works great to give me a chance to backup my jellyfin media. Everything I care about also gets backed up via restic.
Eventually plan to run a build with the Modcase MASS with multiple drives but for now this setup has been working fantastic.
=> More informations about this toot | More toots from radau@lemmy.dbzer0.com
Love the effort you’ve put into this question. You’ve clearly done some quality research and thinking.
When I asked myself this same question a couple of years ago, I ended up just buying a second hand Synology NAS to use alongside my mini-pc. That would meet your criteria, and avoids the (I’m not sure what magnitude) reliability risk of using disks connected over USB. It’s more proprietary than I’d like, but it’s battle tested and reliable for me.
=> More informations about this toot | More toots from thirdBreakfast@lemmy.world
I personally had the best experience with mergerfs (Drives can be any size and can be backuped by snapraid) and an external enclosure up until recently. Unfortunately USB is such a limiting factor because of bandwidth and also latency. I can only realy recommend to get a new cheap Server which has Support for sata If you want any usability while Transferring or moving files.
=> More informations about this toot | More toots from browse@lemmy.specksick.com
How does mergefs compare to btrfs and bcachefs in using multiple partitions?
=> More informations about this toot | More toots from InnerScientist@lemmy.world
Drives connected to usb have an unstable connection in my experience, this is very annoying and gets worse with hubs.
RAIDs reduce the time a system is offline and reduce data loss, if a drive fails and you can afford to wait for the new disk and the backup to restore, and have regular backups that ensure no important data gets lost (though remember the data added between backups may be lost) then you don’t need a RAID.
I don’t use RAIDs cause if my disk fails then I can stomach the 2-4 days it takes to buy a new one and restore the backup
Very important: use S.M.A.R.T and a filesystem with checksums to make sure you’re not backing up corrupted data and know to get a new one
For encryption at rest you may want to look at clevis and tang, though you need a server in your home network for this to work. The client (with clevis) then decrypts the disk at boot if it can reach the server (tang). The server can’t decrypt the data without the client secret and the client can’t decrypt it without the server public key.
Don’t know what your server could be though, maybe a router with custom firmware?
You should also look into cloud storage/rclone, that way you can automate your backups more and reduce the need for manual intervention.
I use rclone and restic to automatically backup my servers daily which takes a few seconds most of the time due to them being incremental backups.
=> More informations about this toot | More toots from InnerScientist@lemmy.world
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters
More Letters
NAS
Network-Attached Storage
Plex
Brand of media server package
RAID
Redundant Array of Independent Disks for mass storage
SSD
Solid State Drive mass storage
[Thread #920 for this sub, first seen 14th Aug 2024, 10:45]
[FAQ] [Full list] [Contact] [Source code]
=> More informations about this toot | More toots from Decronym@lemmy.decronym.xyz
You’ve clearly done your homework, and you’ve gotten a lot of good feedback already, so I’ll just add a few points…
Encryption: If you’re going to bother with encryption, I wouldn’t half-ass it. Why bother at all if you’re fine using auto-decryption or a password that will be guess with any sizeable effort? Just lock it down with a strong password and decrypt/mount the data drive after any reboot; making a shell alias or script for this is trivial. You’re likely not rebooting the server more than once a week anyway.
Budget/Specs: I get the sense you don’t have much budget right now, but knowing your hardware would help in suggesting solutions. Do you have an NVMe slot? What kind of mini-PC case?
Filesystem: For simple storage, this really doesn’t matter and Ext4 will probably be fine. It’s a mature, robust, no-frills filesystem which is perfect for bulk file storage (docs, music, videos, etc.), but Btrfs would be fine too if you want more options.
USB Docking Stations: I’ve had really good experiences with USB docking stations like this one, and I currently use it for attaching my backup HDDs each month. I wouldn’t want to rely on them for realtime data storage, but they do work wonderfully for backups and one-off drive access.
=> More informations about this toot | More toots from thayer@lemmy.ca
For automatically unlock encrypted drives I followed the approach described in …stapelberg.ch/…/2023-10-25-my-all-flash-zfs-netw…
The password is split half in the server itself and half in a file on the web. During boot the server retrieves the second half via http, concatenates the two halves and use the result to unlock the drive. In this way I can always remove the online key and block the automatic decryption.
Another approach that I’ve considered was to store the decryption keys on a USB drive connected with a long extension cable. The idea is that if someone will steal your server likely won’t bother to get the cables too.
TPM is a different beast I didn’t study yet, but my understand is that it protects you in case someone steals your drives or tries to read them from another computer. But as long as they are on your server it will always decrypt them automatically. Therefore you delegate the safety of your data to all the software that starts on boot: your photos may still be fully encrypted at rest so a thief cannot get them out from the disk directly, but if you have an open smb share they can just boot your stolen server and get them out from there
=> More informations about this toot | More toots from lorentz@feddit.it
Hi OP, I am in a similar situation as you were so I wonder which solution you chose at the end and if you are happy with it?
=> More informations about this toot | More toots from scrooge101@lemmy.ml
I chose to continue with my current setup until I get the time and motivation to upgrade.
I will build a new server from scratch. For that, I bought an used mainboard for a few bucks, which has 6 SATA slots.
=> More informations about this toot | More toots from Guenther_Amanita@slrpnk.net This content has been proxied by September (3851b).Proxy Information
text/gemini