sege
Graham Sutherland / Polynomialit is extremely funny to me that a literal trillion dollars funnelled into a cloud provider results in more sev0 / sev1 outages per year than you'd get with a hosting provider run by a polycule out of an apartment block in rural Germany.
Manawyrm | Sarah@gsuberland That's why you just host at the big german hosters, where the catgirls are also in charge of operations and things work :P
Marcus Müller@funkylab @gsuberland If you have an ops team entirely made out of https://xkcd.com/705/ people... That'll do the trick :)
ARGVMI~1.PIF
Andrew Zonenberg@gsuberland The only significant downtime I've had in the last year or two for my offsite backup server (currently hosted by a catgirl polycule out of a house across the water from Seattle) was when said polycule bought the house, moved in together, and physically relocated the server from the rented house it had been living in previously.
That, and I think there was some issues due to a fiber cut that affected the whole county earlier in the year. But the North American Fiber-Seeking Backhoe isn't choosy about its victims.
Mark T. Tomczak@azonenberg @gsuberland Yeah, if anything, it's fun sometimes to reflect on how infrequently the telephonic fabric goes down.
At least we seem to have solved that problem pretty well.
scunneen (he/him) 🚋@mark @azonenberg @gsuberland The telephone service in my (suburban NJ) town has become pretty unreliable in the last few months— based on something Verizon support told me they’re keeping old parts in service till they break, probably thinking that most people don’t care about landlines anymore.
Jean-Baptiste "JBQ" Quéru@gsuberland nobody ever got fired for hosting on AWS
Orb 2069... Yet.
Jonty Wareing@gsuberland I mean *sure* but also most tiny providers rather balk at requests like "I would like to replicate 3pb of data to you tomorrow would that be ok kthx". Nobody running tiny easily movable things is in us-east-1 anymore.
@jonty I know, I know, apples to oranges :P
@gsuberland More like grapes to moons
@jonty also yes
although on this note I recently priced up a 100PiB rack (complete with redundant head nodes) and it wasn't as ridiculous as I was expecting. within "handful of friends deciding to set up a backup provider" distance. drive density per £ has been skyrocketing.
@jonty although I suspect the real wallet-stinger would be peering somewhere that can meaningfully support such large transfers
bob@gsuberland @jonty you can get 1U in HE FMT2 for like $100/mo and then you can peer at SFMIX (I don't know how much the cross-connect is though)
Richard "RichiH" Hartmann@gsuberland @jonty truck full of disks has the highest bandwidth. Always has, always will. At that amount of data, you'd likely drive over at least the initial sync
Cavedale Rhônes
BillRemember Kermit? Back in the 80s, it's documentation said never underestimate the bandwidth of a station wagon (What's that Uncle Bill?) full of magnetic tape. I think high density was 1600bpi back then.
Erin 💽✨@gsuberland @jonty transit has gotten cheaper, depends upon bandwidth requirements. Also peering cost is heavily location dependent and who you need to receive data from
@gsuberland @jonty (which is to say: your customers use Telekom or Orange? You're fucked and the only thing saving you from being uberfucked is the fact that your DTAG customers are stuck on VDSL)
@gsuberland @jonty ehh -- at least in Europe there are some good locations. apart from some edgecases like Deutsche Telekom, it's managable.
@gsuberland Ooooh nice!
I am still a bit sad that I had to turn down the COLOSSAL TAPE ROBOT not because I didn't have a place to put it, but because they refused to give us the tapes and replacing them would have been eyewatering.
@jonty please tell me that it looked something like that scene from Hackers where they're fighting over the TV station
@gsuberland I will neither confirm or deny that the main reason I wanted a tape robot was to re-enact that scene
@jonty would've been incredible
@gsuberland You say that like I've stopped trying to acquire a tape robot
@gsuberland This is your fault: https://chaos.social/@jonty/115408579704684143
Jonty Wareing (@[email protected])
Tape Robot Wars: Two teams battle to retrieve and insert the correct LTO cartridge using custom-built robot arms while Craig Charles narrates the action
@jonty someone in the extended EMF crew *must* have an in with Craig Charles, surely
@gsuberland Not that I've discovered. Robert yes, but he keeps turning us down.
@jonty ah damnit. isn't Dara Ó Briain doing the new series of Robot Wars? he's a major nerd, I'd bet he'd be an easier sell on the concept.
@gsuberland Graham how could you give me more projects like this
tef
Garrett Wollman@jonty @gsuberland On a call last week, our backup person said that a ten-pack of LTO9s is like $24,000 and now I'm not quite so surprised that people are buying disks rather than tapes, because that's more expensive than the same volume of 24T CMR HDDs and slower.
Le personified devil@wollman Honestly I'm not sure what ur backup person is doing, but a single LTO-9 cartridge from Hewlett Packard costs less than 100$. (https://buy.hpe.com/us/en/storage/storage-media/tape-media/lto-ultrium-cartridges/hpe-lto%E2%80%919-ultrium-45tb-rw-data-cartridge/p/q2079a)
@personifieddevil It may be that I misunderstood. Perhaps she was talking about ten *packs*, for the robot, rather than ten individual tapes. That would make a lot more sense in terms of price per TB.
mikechislett@jonty @gsuberland
Tempting.
How tied to a particular generation of LTO are you for ridiculous cupboard storage?
(Also I assume Basil still has those Petabytes, I have no idea where they went after we put them in the car)
Tempting.
How tied to a particular generation of LTO are you for ridiculous cupboard storage?
(Also I assume Basil still has those Petabytes, I have no idea where they went after we put them in the car)
Sven Slootweg, low-spoons mode ("still kinky and horny anyway")@jonty @gsuberland I have bad news for you about all the tech startups whose entire business could fit in a moderately sized postgres database
@joepie91 @gsuberland What you mean all of them
Charlotte 
/Cinny 
@joepie91 @gsuberland @jonty i think there’s also a bunch of exceptions to behavior that exclusively apply to us-east-1
at which point i wonder if those exceptions may be why some people might choose to go there even though the exceptions almost certainly just are due to some very well-paying customers doing it wrong™
mathew
neville park@gsuberland what's their fediverse instance
Jernej Simončič �@gsuberland I had a long outage of my server yesterday, but that was because I replaced the motherboard, then foolishly updated BIOS, which first resulted in the PCIe x1 slot (where I stuffed the graphic card) getting disabled, and then BIOS not accepting keyboard input to confirm that I'm fine with TPM being reset (had to connect the keyboard directly instead of going through the KVM).
schrotthaufen@gsuberland @cstross Rural Germany has maybe DSL 16000, if you’re lucky 😭
@schrotthaufen @cstross the catgirl polycule will find that one place in Meck-Pom that happens to have a fiber trunk running past it
@gsuberland @cstross Sounds plausible
haihappen@gsuberland @schrotthaufen @cstross there are a lot of small-ish regional fibre companies now in Germany. They have mixed ... Success. Meaning: everybody wants fibre, and get rid of their monopolistic supplier, but only few small cities have been connected. Mainly there are roadblocks in permits (local gov infra perms are a nightmare), and resources: they got hit by inflation and labor shortages.
I signed up for fibre 3y ago. Still on shitty DSL.
@gsuberland @schrotthaufen @cstross if there was fibre here, I would invest into a small scale local data center, just so there was an option for small businesses that is not the "it pays the fees or it gets the hose again" big cloud providers.
@haihappen @gsuberland @cstross Yeah, the permit process is mind boggling. Getting Passierschein A38 is easier…
Björn 🌸 StarkimarmSame, but that's on fucking Deutsche Glasfaser, who tried to bite off more than they can chew. My family 2 towns over have fibre by their local utilities provider (for less money)..
moanos@gsuberland @schrotthaufen @cstross There were webserversrunning in forest occupations in germany, powered by solar+directional antenna
silverwizard@gsuberland The polycule in rural Germany probably has a more skilled staff though
webhat@gsuberland I think it was a marketing stunt by incident.io to promote that the sev0 conference is in London in 3 days
anna, a criticality incident at the gender accelerator@gsuberland yeah the german polycule doesn't have AI write production code
F4GRX Sébastien@anna @gsuberland it was DNS they said. Lol
@f4grx @gsuberland ai powered dns updates
@anna @gsuberland This would not surprise me...