What is the largest file transfer you have ever done?
https://startrek.website/post/13578615
=> More informations about this toot | More toots from data1701d@startrek.website
@data1701d downloading forza horizon 5 on Steam with around 120gb is the largest web-download, I can remember. In LAN, I’ve migrated my old FreeBSD NAS to my new one, which was a roughly 35TB transfer over NFS.
=> More informations about this toot | More toots from Larvitz@burningboard.net
How long did that 35TB take? 12 hours or so?
=> More informations about this toot | More toots from data1701d@startrek.website
@data1701d little bit over 4 days with average speed of around 850mbps
=> More informations about this toot | More toots from Larvitz@burningboard.net
Entire drive/array backups will probably be by far the largest file transfer anyone ever does. The biggest I’ve done was a measly 20TB over the internet which took forever.
Outside of that the largest “file” I’ve copied was just over 1TB which was a SQL file backup for our main databases at work.
=> More informations about this toot | More toots from fuckwit_mcbumcrumble@lemmy.dbzer0.com
+1
From an order of magnitude perspective, the max is terabytes. No “normal” users are dealing with petabytes. And if you are dealing with petabytes, you’re not using some random poster’s program from reddit.
For a concrete cap, I’d say 256 tebibytes…
=> More informations about this toot | More toots from cbarrick@lemmy.world
brother?..
=> More informations about this toot | More toots from cactusupyourbutt@lemmy.world
10TB is child’s play
=> More informations about this toot | More toots from possiblylinux127@lemmy.zip
I don’t remember how many files, but typically these geophysical recordings clock in at 10-30 GB. What I do remember, though, was the total transfer size: 4TB. It was kind of like a bunch of .segd files (geophysics stuff), and they were stored in this server cluster that was mounted in a shipping container, and some geophysics processors needed it on the other side of the world. There were nobody physically heading in the same direction as the transfer, so we figured it would just be easier to rsync it over 4G. It took a little over a week to transfer.
=> More informations about this toot | More toots from neidu2@feddit.nl
4G?! That strikes fear into my heart!
=> More informations about this toot | More toots from data1701d@startrek.website
The alternative was 5mbit/s VSAT. 4G was a luxury at that time.
=> More informations about this toot | More toots from neidu2@feddit.nl
At the rates I’m paying for 4G data, there are very few places in the world where it wouldn’t be cheaper for me to get on a plane and sneakernet that much data
=> More informations about this toot | More toots from RegalPotoo@lemmy.world
We have DBs in the dozens of TB at work so probably one of them
=> More informations about this toot | More toots from ryannathans@aussie.zone
~15TB over the internet via 30Mbps uplink without any special considerations. Syncthing handled any and all network and power interruptions.
=> More informations about this toot | More toots from avidamoeba@lemmy.ca
How long did that take? A month or two? I’ve backfilled my NAS with about 40 TB before over a 1 gig fiber pipe in about a week or so of 24/7 downloading.
=> More informations about this toot | More toots from pete_the_cat@lemmy.world
Yeah, something like that. I verified it it with rsync after that, no errors.
=> More informations about this toot | More toots from avidamoeba@lemmy.ca
I think it’s crazy that not that long ago 30mbps was still pretty good, we now have 1gbps+ at residential addresses and it fairly common too
=> More informations about this toot | More toots from independantiste@sh.itjust.works
Yeah, I also moved from 30Mb to 700Mb recently and it’s just insane. It’s also insane thinking I had a symmetric gigabit connection in Eastern Europe in the 2000s for fairly cheap. It was Ethernet though, not fiber. Patch cables and switches all the way to the central office. 🫠
=> More informations about this toot | More toots from avidamoeba@lemmy.ca
We might pay some of the most expensive internet in the world in Canada but at least we can’t fault them for providing an unstable or unperformqnt service. Download llama models is where 1gbps really shines, you see a 7GB model? It’s done before you are even back from the toilet. Crazy times.
=> More informations about this toot | More toots from independantiste@sh.itjust.works
I should have know that the person on the internet noting 30Mbps was pretty good till recently is a fellow Canadian. 🍁 #ROBeLUS
=> More informations about this toot | More toots from avidamoeba@lemmy.ca
Fiber is so nice
=> More informations about this toot | More toots from possiblylinux127@lemmy.zip
I’ve got symmetrical gigabit in my apartment, with the option to upgrade to 5 or 8. I’d have to upgrade my equipment to use those speeds, but it’s nice to know I have the option.
=> More informations about this toot | More toots from Confused_Emus@lemmy.dbzer0.com
Currently pushing about 3-5 TB of images to AI/ML scanning per day. Max we’ve seen through the system is about 8 TB.
Individual file? Probably 660 GB of backups before a migration at a previous job.
=> More informations about this toot | More toots from psmgx@lemmy.world
I mean dd claims they can handle a quettabyte but how can we but sure.
=> More informations about this toot | More toots from bulwark@lemmy.world
dd can’t really handle quettabytes! GNU has taken us all for fools! Alert the masses! Wake up sheeple!
=> More informations about this toot | More toots from data1701d@startrek.website
=> More informations about this toot | More toots from possiblylinux127@lemmy.zip
Not that big by today's standards, but I once downloaded the Windows 98 beta CD from a friend over dialup, 33.6k at best. Took about a week as I recall.
=> More informations about this toot | More toots from Davel23@fedia.io
Yep, downloaded XP over 33.6k modem, but I’m in NZ so 33.6 was more advertising than reality, it took weeks.
=> More informations about this toot | More toots from absGeekNZ@lemmy.nz
I remember downloading the scene on American Pie where Shannon Elizabeth strips naked over our 33.6 link and it took like an hour, at an amazing resolution of like 240p for a two minute clip 😂
=> More informations about this toot | More toots from pete_the_cat@lemmy.world
Totally worth it.
=> More informations about this toot | More toots from boobyboy69@lemmy.world
In similar fashion, downloaded dude where’s my car, over dialup, using at the time the latest tech method - a file download system that would split the file into 2mb chunks and download them in order.
It took like 4 days.
=> More informations about this toot | More toots from 50MYT@aussie.zone
i’ve transferred 10’s of ~300 GB files via manual rsyncs. it was a lot of binary astrophysical data, most of which was noise. eventually this was replaced by an automated service that bypassed local firewalls with internet-based transfers and aws stuff.
=> More informations about this toot | More toots from Llituro@hexbear.net
As a single file? Likely 20GB iso.
As a collective job, 3TB of videos between hard drives for Jellyfin.
=> More informations about this toot | More toots from slazer2au@lemmy.world
Largest one I ever did was around 4.something TB. New off-site backup server at a friends place. Took me 4 months due to data limits and an upload speed that maxed out at 3MB/s.
=> More informations about this toot | More toots from Yeahboiiii@lemm.ee
Around 15 TB migrating to a new NAS.
=> More informations about this toot | More toots from narc0tic_bird@lemm.ee
You should ping CERN or Fermilab about this. Or maybe the Event Horizon Telescope team but I think they used sneakernet to image the M87 black hole.
Anyway, my answer is probably just a SQL backup like everyone else.
=> More informations about this toot | More toots from ShittyBeatlesFCPres@lemmy.world
I routinely do 1-4TB images of SSDs before making major changes to the disk. Run fstrim on all partitions and pipe dd output through zstd before writing to disk and they shrink to actually used size or a bit smaller. Largest ever backup was probably ~20T cloned from one array to another over 40/56GbE, the deltas after that were tiny by comparison.
=> More informations about this toot | More toots from seaQueue@lemmy.world
In grad school I worked with MRI data (hence the username). I had to upload ~500GB to our supercomputing cluster. Somewhere around 100,000 MRI images, and wrote 20 or so different machine learning algorithms to process them. All said and done, I ended up with about 2.5TB on the supercomputer. About 500MB ended up being useful and made it into my thesis.
Don’t stay in school, kids.
=> More informations about this toot | More toots from Neuromancer49@midwest.social
You should have said no to math, it’s a helluva drug
=> More informations about this toot | More toots from mEEGal@lemmy.world
m.youtube.com/watch?v=6xi2Bl50XPc
=> More informations about this toot | More toots from data1701d@startrek.website
golden 😂😂
=> More informations about this toot | More toots from mEEGal@lemmy.world
Local file transfer?
I cloned a 1TB+ system a couple of times.
As the Anaconda installer of Fedora Atomic is broken (yes, ironic) I have one system originally meant for tweaking as my “zygote” and just clone, resize, balance and rebase that for new systems.
=> More informations about this toot | More toots from boredsquirrel@slrpnk.net
Today I’ve migrated my data from my old zfs pool to a new bigger one, the rsync of 13.5TiB took roughly 18 hours. It’s slow spinning disks storage so that’s fine.
The second and third runs of the same rsync took like 5 seconds, blazing fast.
=> More informations about this toot | More toots from tedvdb@feddit.nl
I’m currently in the process of transferring about 50 TB from one zpool to another (locally), so I can destroy and recreate it.
=> More informations about this toot | More toots from pete_the_cat@lemmy.world
I have to copy ~8 TiB of backup files twice a year.
=> More informations about this toot | More toots from Laborer3652@reddthat.com
I’ve migrated petabytes from one GPFS file system to another. More than once, in fact. I’ve also migrated about 600TB of data from D3 tape format to 9940.
=> More informations about this toot | More toots from d00phy@lemmy.world
I’ll let you know… If it finishes.
=> More informations about this toot | More toots from MajorHavoc@programming.dev
Professionally, Boeing near Seattle moves mind-boggling data. Personally,1500+ terabytes in a single 6 day jag, aboard a 29’ sailboat running off an inverter and 13 year old Dell hardware, completely off grid on anchor. It was a massive rescue operation, I did what I could with what I had where I was at.
I didn’t sleep well.
=> More informations about this toot | More toots from nobleshift@lemmy.world
I once robocopied 16tb of media
=> More informations about this toot | More toots from averyminya@beehaw.org
I obviously downloaded a car after seeing that obnoxious anti-piracy ad.
=> More informations about this toot | More toots from Urist@lemmy.ml
I once deleted an 800 gb log file, does that count
=> More informations about this toot | More toots from pixeltree@lemmy.blahaj.zone
Depends, did you send it to the trash can first?
=> More informations about this toot | More toots from Loulou@lemmy.mindoki.com
Nah, rm’d on shudders Oracle linux
=> More informations about this toot | More toots from pixeltree@lemmy.blahaj.zone
I did 100TB, 100 streams of 1TB, all simultaneous with rsync
=> More informations about this toot | More toots from Nomecks@lemmy.ca
I transferred my entire NAS storage, which includes all of my backups, cloud files, my family’s backups, and my… Linux ISOs. That was about 12TB.
=> More informations about this toot | More toots from hperrin@lemmy.world
I’ve done a 1PB sync between a pair of 8-node SAN clusters as one was being physically moved since it’d be faster to seed the data and start a delta sync rather than try to do it all over a 10Gb pipe. M
=> More informations about this toot | More toots from ramble81@lemm.ee
I think 16 terabytes? Might have been twelve. I was consolidating a bunch of old drives and data into a nas for a friend. He just didn’t have the time, between working and school and brought me all the hardware and said “go” lol.
=> More informations about this toot | More toots from southsamurai@sh.itjust.works
I once abused an SMTP relay (my own) by emailing Novell a 400+ MB memory dump. Their FTP site kept timing out.
After all that, and them swearing they had to have it, the OS team said “Nope, we’re not going to look at it”. Guess how I feel about Novell after that?
This was in the mid-90’s.
=> More informations about this toot | More toots from Trigger2_2000@sh.itjust.works
Well, at least they were being on-brand. 😅
=> More informations about this toot | More toots from data1701d@startrek.website
Manually transferred about 7TBs to my new Rpi4 powered NAS. It took a couple of days because I was lazy and transferred 15 GBs at a time which slowed down the speed for some reason. It could handle small sub 1 GB files in half a minute otherwise.
=> More informations about this toot | More toots from Someonelol@lemmy.dbzer0.com
Could the slowdown be down to HDDs that cache on a section of - I think it’s single layer? - and slowly rewrite that cache onto the denser (compound layer?) storage?
=> More informations about this toot | More toots from milicent_bystandr@lemm.ee This content has been proxied by September (3851b).Proxy Information
text/gemini