RmDebArc_5Save animals, push to prod
Jim Spath162335,162335,test_Eutaw @ Lumbard,,39.287570867,-76.62116607,,0,,0,
Musing Mouse“deliberately obscured payment-blocking capabilities being added to absolutely critical government infrastructure by an inexperienced developer with minimal oversight."
A 25-Year-Old Is Writing Backdoors Into The Treasury’s $6 Trillion Payment System. What Could Possibly Go Wrong?
Just months after we learned Chinese hackers had compromised US telecom systems through government-mandated backdoors, an inexperienced developer from Musk’s DOGE unit is pushing untested cod…
Liz Fong-Jones (方禮真)Testing in production requires us to organize our infra blast radius, and to have a blameless culture instead of risk aversion.
Infrastructure as code is an abstraction... not everything maps 1:1 but it's close! [fin] #TestInProduction
For A/B testing, you need to check whether you are meeting your SLOs, as well as being cheap enough...
Write down your hypotheses about how you'll improve SLOs or decrease cost with your changes. Then test it. #TestInProduction
Put higher risk tolerance apps in a taint group.
Once new VPC is spun up and all the instances live in it, then we can tear down the old one. [ed: I... wonder how one does this without AWS deciding it's gonna delete/recreate once network[1] becomes network[0]] #TestInProduction
We also need to have canaries that we can verify that our networks are functional and can connect to/from the outside... before we migrate VMs into the subnets. #TestInProduction
One way of doing this is to create more than one copy of the network, using a toggle variable to control whether 1 or 2 copies exist, and then the second nested copy has different properties.
[ed: we love this technique and use it at Honeycomb too!] #TestInProduction
So now, having rolled out 35% of all traffic, they were able to finish everything to 100% having enough confidence they'd caught everything. 15-month project to the finish line in the last 2 days!
Make sure to clean up migration intermediate state debt! [fin] #TestInProduction
If you can shift work left and use a staging environment, it might help you.
But, with a network, says @joatmon08, there's no practical way to simulate a full production scale network.
What can we apply? Feature flags. Preserve state, and test rollforward/back #TestInProduction