b33rdyApr 2, 2024
Would you use a k8s service for a highly complex platform with billions of users where you were not permitted access to the api-service? I.e. no kubectl or other tooling of your choice, only a defined means of deployment/ pod log access? #k8s #kubernetes #devops #gitops
Show threadTim Stoop Apr 2, 2024
@b33rdy
That could be a proper solution, but it depends so much on the entire story. What's the level of support and observability? Knowledge and experience of the devs? Type of app? Not an automatic "no" without more context for me, at least.
Show threadb33rdyApr 2, 2024

@timstoop great question.

Some devs aren’t concerned with hosting - fire and forget. Others have worked with clusters of over 50k nodes.

We have a high degree of engineering skills.

The observability solution was created to look after the infra. We haven’t seen it from a dev perspective yet.

The app is more of a platform. Game server orchestration. Bare metal management. Invoicing. Service desk etc etc. it’s vast. Some 70/80 unique service types (at the moment).

Show threadb33rdyApr 2, 2024
@timstoop It’s for an SME that owns 70k~ server assets globally, including 4 owned data centres.
Show threadTim Stoop Apr 2, 2024
@b33rdy
And your question is about being a developer working in such a situation? Or a development company as a whole? In either case, though, it depends a lot on trust. Not having direct API access makes sense in a lot of situations, but it depends as well on how well the actual ops respond to any issue and whether devs have enough insight into running their code to see what issues there are. IMHO, ofc. I personally would hate it, but I'm a k8s ops person 😅
Show threadTim Stoop Apr 2, 2024
@b33rdy
That said, principle of least privilege is not just a security tool, it's also a way to separate concerns. There are lots of situations in which it makes perfect sense for devs to just not be concerned about the actual running of the platform underlying the app, especially if it's used as a platform (as opposed to a framework, where you actually interact with the API from the app).
Show threadb33rdy

@timstoop some great points Thankyou.

There are a lot of trust issues at play. From our (dev) perspective we don’t have a lot of faith in the ops teams ability to run the platform.

So we (dev) would prefer more access (r/o is fine) to enable better tooling/ alerting/ o11y in general rather than it being dictated.

Apr 2, 2024 at 8:58amWeb
Show threadTim Stoop Apr 2, 2024
@b33rdy
You said the magic words there that are sure to make this a bad experience at best and a failure at worst. Trust is key. But not considering dev and ops to be part of the same team is a huge red flag to me. Sounds like silos and those always chafe up to the point of bleeding if you cannot add a proper padding in between. Or better yet, remove the walls. Again, just imho.
Show threadTim Stoop Apr 2, 2024
@b33rdy
You wanting API access is a symptom to a deeper problem that needs to be solved. Especially on your scale. Assuming remote work, devs and ops should be in the same channel. If on-premise, ops should have (at least!) a representative present at stand ups. Have them hear the issues and take ownership, if they are proper ops people, they care about your issues. But make sure they are solvable. "We don't trust you, period" is unsolvable.
Show threadb33rdyApr 2, 2024

@timstoop new CTO is trying to do exactly that.

When the internal solution was made - dev weren’t included in the requirements gathering and aren’t down as stakeholders.

It’s super odd.

Show threadTim Stoop Apr 2, 2024
@b33rdy Hold on tight to that one, then. Sounds like they know what they're doing, the new CTO! 😉
Show threadb33rdyApr 2, 2024
@timstoop agreed! Thanks for the chat.