skomposzczetJul 3, 2023

How do social media companies like twitter or youtube not run out of space for posts?

https://vlemmy.net/post/461587

How do social media companies like twitter or youtube not run out of space for posts? - VLemmy

I know memory is fairly cheap but e.g. there are millions of new videos on youtube everyday, each probably few hundred MBs to few GBs. It all has to take enormous amount of space. Not to mention backups.

Show threadassemblyJul 3, 2023
It’s the same story with AWS as well. They use vast amounts of storage and leverage different tiers of storage to get the service they want. It’s funny but they have insane amounts of SD cards ( cheapest storage available at the size) and use that for some storage and just replicate things everywhere for durability. Imagine how small 256GB SD cards are and they you have hardware to plug-in 200 of them practically stacked on top of each other. The storage doesn’t need to be the best, it just needs to be managed appropriately and actively to ensure that data is always replicated as devices fail. That’s just the cooler tier stuff. It gets complex as the data warms.
Show threadLostXORJul 3, 2023
SD cards? I'm very skeptical. Do you have a source?
Show threadasteroidrainfall
Yeah this seems false. SD cards are unreliable, hard to keep track of, and don’t actually store that much data for the price. I do think they use tapes though to store long term, low traffic data.
Jul 3, 2023 at 11:34pmWeb
Show threadzalackJul 4, 2023

We use LTO tapes in Hollywood to back up raw footage; it wouldn't surprise me if AWS uses tapes for glacier.

I got a tour of Iron Mountain once (where we sent tapes for long term archival). They had a giant room with racks and racks of LTOs, and robot on rails that would make copies of each tape at regular intervals to keep the data from corrupting. Super interesting.

Show threadLostXORJul 4, 2023
That sounds really cool! I wonder what sort of redundancy they have in case a tape gets damaged or corrupted.