Neil CraigJul 4, 2024
Here's a few of the prep tasks we've done for the UK General Election in/around our team:
- Add extra CDN/peering capacity
- Ensure personal & machine credentials & TLS cert work & won't expire
- Raise service quotas for safety
- Check that deployment pipeline are clean/deployable
- Ask 3rd parties to be extra-vigilant on change requests
- Rehearse failovers to backup systems
- Make sure all logging/metrics are working correctly
Probably mainly/all pretty obvious but maybe useful.
#BBC #GE2024
Show threadNeil Craig
We're expecting a spike on the web and media streams at 10pm BST when the exit poll results are communicated.
Unsure quite how large the spikes will be...that'll depend very much on how close to expectation/prediction the exit polls results are.
There's a good number of us working to cover regular ops but also being available for additional checks/monitoring and escalation paths if needed.
I hope everyone who's working on #GE2024 (and anything else this evening/today) is doing well. 🙌🏻
Jul 4, 2024 at 8:26pmWeb
Show threadNeil CraigJul 4, 2024
Just noticed we're at ~1.3k RPS from BBC staff and in-house, on-network monitoring...
Lots of eyes on for sure.
Show threadNeil CraigJul 4, 2024
The first main Breaking News Event (BNE) notification just went out...That caused roughly a 2x traffic spike on www.bbc.co.uk & www.bbc.com but nothing to worry about, waaaaay within bounds.
Let's see what happens when the exit poll results are communicated out.
Show threadNeil CraigJul 4, 2024
And here's the exit poll: https://www.bbc.co.uk/news/live/cn09xn9je7lt
General election live: Keir Starmer names new cabinet, with Reeves as chancellor and Rayner as deputy PM

The prime minister appoints his top team after pledging “the work of change begins immediately”.

BBC News
Show threadNeil CraigJul 4, 2024
The Breaking News Event notification caused roughly a 4x spike versus BAU traffic. Again, still well within the bounds we're comfortable with.
Show threadNeil CraigJul 4, 2024

There's also a new feature this election: "Camera at every count" which you can see here if you're able to access iPlayer: https://www.bbc.co.uk/iplayer/episodes/m002177v/watch-your-count-live

The tech behind it is really interesting, essentially it uses a system developed to take advantage of how good mobile phones & networks are these days. We have a live stream from every one of the 385 election counts in the UK & there's an internal representation of them as a mosaic of mosaics.

I wish I could share it but I can't. Sorry.

Watch Your Count Live

Watch as votes are counted across the UK at over 370 locations

BBC iPlayer
Show threadNeil CraigJul 4, 2024
Fair few people on the #GE2024 live page (https://www.bbc.co.uk/news/live/cn09xn9je7lt).
I've not kept track of this much, I've been focussing mainly on uptime and performance monitoring but it's been in the region of 250k+ people the last few times I saw it. Doubtless higher when the Breaking NEws Event notification was sent.
General election live: Keir Starmer names new cabinet, with Reeves as chancellor and Rayner as deputy PM

The prime minister appoints his top team after pledging “the work of change begins immediately”.

BBC News
Show threadNeil CraigJul 4, 2024
First #GE2024 result in, Sunderland South - Labour win.
https://www.bbc.co.uk/news/election/2024/uk/results
UK election results 2024 | Constituency map

Get the latest news and election results in the 2024 General election from BBC News

BBC News
Show threadmarkwalkerJul 5, 2024
@tdp_org Your grafana stats will be interesting reading tomorrow!
Show threadNeil HenningJul 4, 2024
@tdp_org loving these updates man! Super interesting.
Show threadDaniel HardyJul 4, 2024
@tdp_org do the bbc have the exit poll broken down by constituency anywhere?
Show threadDr. David McBride (dwm)Jul 4, 2024
@tdp_org Just a small thank you for all of your efforts for an apparently effortless bulletproof service. :)
Show threadNeil CraigJul 4, 2024
@dwm Cheers! Much appreciated, we know not everyone loves what we do but we give it our best every time and it means a lot when people value our work 🙌🏻
Show threadDr. David McBride (dwm)Jul 4, 2024
@tdp_org It's one of those things when maintaining infrastructure — if people notice you, something's gone badly wrong. <3
Show threadNeil CraigJul 4, 2024
@dwm 100%
Our department lead always says "don't let the tech be the story".
Show threadNashaJul 4, 2024
@tdp_org well now you're just making me jealous 😂
Show threadNeil CraigJul 4, 2024
@nasha 🤣 I really wish I could screenshot and share it. It's quite a thing!
I hope you're doing well BTW 🙌🏻
Show threadGlenjaminJul 4, 2024
@nasha @tdp_org out of interest, if all 385 streams are available, what’s the rule that stops you sharing a picture of all of them on one screen?
Show threadCatButtesJul 4, 2024
@tdp_org Was that the wall of videos shown briefly in the election results intro on BBC 1 while they were filling space before they could announce the exit polls? It looks pretty cool and I would love to have a peek at what is powering it in the background!
Show threadNeil CraigJul 4, 2024
@catbuttes I didn't see it but it sounds very much like it.
Maybe I'll ask if the folks involved would write it up as a blog post.
The thing I really like about it is that it's *very* pragmatic, task-focussed tech. Gets the job done most of the time with a lot less resources that a super-reliable system.
Show threadCatButtesJul 4, 2024
@tdp_org I would love to read a write up of everything you guys have going on behind the scenes to make sure this all works. It all sounds incredibly interesting (and I would probably look for any ideas I can poach for my work!)
Show threadDon ThompsonJul 4, 2024
@tdp_org This doesn't look right...
Show threadDave HollandJul 4, 2024
@tdp_org Brilliant idea!
Show threadAJul 5, 2024

@tdp_org this video popped up, looks like a great idea.

https://www.tiktok.com/@lauragarciarb/video/7388020057890180385?_t=8

TikTok - Make Your Day

Show threadHughJul 4, 2024
@tdp_org Shame we don't get an absolute scale on that graph 😛
Show threadMalissa LeeJul 5, 2024
@tdp_org iPlayer Mobile had quite a jump during the exit poles as well looking at our dashboards.
Show threadPepperstacheJul 4, 2024
@tdp_org is there a page about the methodology of the exit poll? Just wondered if the forecast is based on national swing
Show threadNeil CraigJul 4, 2024
@qux2000 I don't know the detail on that, sorry - it's outside of what I do. It is/was conducted by Ipsos though so I'd hope they publish methodology. Maybe worth a look on their website. I'd assume it's sampled and upscaled but quite how, I don't know.
Show threadPepperstacheJul 4, 2024

@tdp_org I found this after the toot, fairly informative https://www.bbc.co.uk/news/articles/c06k8ge1ng7o

Appreciate the effort keeping the site up, must be a hell of a night!

How the BBC reports the UK general election 2024

How the BBC covers elections and an explanation of some of terms used.

BBC News
Show threadDavid OakleyJul 4, 2024
@tdp_org I saw an article by @BBCRD a while back about reducing the transmission delay when showing sport over IP... We need that for election nights!
Show threadTonyJul 4, 2024
@tdp_org ahh, I miss those days!
Show threadArch Jul 4, 2024
@tdp_org There must be some fun graphs flying around right now 😅 Good luck!
Show threadJames SewardJul 4, 2024
@tdp_org Hope you have an uneventful night :)
Show threadNeil CraigJul 4, 2024
@jamesoff Thank you. Those are definitely the best kind of nights...tech should be boring for events like this eh? 🤣
Show threadJames SewardJul 4, 2024
@tdp_org definitely, events should be non-events! Sounds like you’ve got your bases covered though :)
Show threadNeil CraigJul 4, 2024
@jamesoff 🤞🏻we've remembered everything!
Events like this really remind you how much of what goes on relies on networking/internet these days...
Show threadJames SewardJul 4, 2024
@tdp_org sometimes miss stuff like this… but I don’t miss holding the pager ;)
Show threadMartin PughJul 4, 2024
@tdp_org fingers crossed for a completely uneventful evening.
Show threadJosh Jul 4, 2024
@tdp_org hugops!!
Show threadJan Lehnardt Jul 4, 2024
@tdp_org all the best 🤞
Show threadNeil CraigJul 4, 2024
@janl Cheers! I think we're firmly in the "enthusiasts zone" now...given the exit poll results were largely as expected. Assuming no major surprises in the morning, I'm hoping for an uneventful day! 🤞🏻