I added a mistake tracker to my ai-tracker. It’s not every mistake. Nobody has time for that.Trying to reduce log bloat and pinpoint patterns. Model depredation seems to coincide with nearing end of plan, time of day, and announcement of a new model.
https://github.com/2ndSightLab/ai-tracker/blob/main/mistake-tracker.md
I had something happen which is a real problem if you are running AI 🤖 agents to write code on the fly. It would be difficult to determine and deal with.
The agent was writing tests and it wrote some code that would produce an eternal loop. It was running the same code over and over and didn’t stop.
A lot of times I’ve seen the agent put a time limit on what they are doing so that doesn’t happen but in this case it didn’t.
So from the outside, how can the system monitoring know it is an eternal loop vs. something that is processing a large number of nodes? How does it know when to kill it vs. just let it run? What if you try to set some arbitrary limit for loops and kill the agent if you can somehow detect it?
Are there any costs involved? If a loop runs forever it may not kill the system. Some programs are designed as forever loops. You would be paying for computer time that is doing nothing on AWS in that scenario
This is a really interesting problem and I’d like to hear how it gets solved at some point.
There’s a gadjillion topics I could write about in these fixed issues I’m making notes on and improvements needed in models before they could ever approach being an actual software architect.
ai-tracker/fixed.md · GitHub · 2ndSightLab/ai-tracker
https://github.com/2ndSightLab/ai-tracker/blob/main/fixed.md
I’ll write about the architecture AI agents struggle with later on blog. But basically this solves one of the problems with AWS control tower. More later.
Last night I just moved the menus to a new project and started over on that code. At some point the two project agents were fighting to use their own code instead of using shared. Kind of like some developers I have worked with. Ok I’ve done it. 😆
I want to take this further but more on that later. Now I have XML driven menus. Yes intentionally chose that. Wrote explicit instructions how the menus should be implemented with file names in the README.
This smaller focused context helps the agent work better on a smaller scope. And yet it didn’t use the same construct for all the menus so I had to have it rewrite some unmanageable. Now all the code is structured the same way, it’s easy to insert new menu items and what actions should be taken when a number is selected.
Also, I added the ability to add and remove organizations, because I plan to create a test organization and my live organization.
In addition the environments and their resource menu. Each environment type is in an XML file. Easier to add and remove new environment types and define what is deployed in them be defining the menu for each environment and pointing to the correct deployment file for each menu item.
I had to be very explicit about defining for the model how to reduce and reuse code. It cannot seem to figure out the construct using sourced files and individual actions in individual files to keep the code easy to follow and reduce breakage. If the program is updating the file for one action it won’t break another.
So I’m getting closer to being able to completely configure an organization and the environments I want to deploy in a way that is adaptable to any organization.
The thing the agents were fighting over was how to create and parse XML files. I also figured out one was using xmllint shell. I removed that but need to evaluate that more later.
I also explained in the fixed notes why I’m restructuring the organization a bit to make it easier to restrict different environments to limited regions and lock regions down when not making changes. The AWS services will still work but no user actions are allowed when an environment is locked.
Last night included a fair amount of thinking time so the resource needle didn’t move too much. I’ve used 50% of my $200 plan since the begin of the month but with smaller code base and locking down some code from incorrect changes hopefully will spin less and figure things out more easily.
The benefit of doing this is that I could start over and redo the menus without messing up my old code. The other issue is that if I revise all the environments before I remove things in the existing environments that aren’t supposed to be there I won’t be able to delete them.
Right now I can simultaneously test both menus until I am sure the new ones work and I can work on creating new environments the way I want and cleaning up resources. I should probably make sure I can clean up any resources that shouldn’t be in an environment anyway.
Managing these things is not simple! Using a flexible approach is key.
I’ve set up budgets for all my AWS accounts using my bootstrap script and you can now monitor those through the AWS cost management dashboard.
https://aws.amazon.com/about-aws/whats-new/2026/05/monitor-aws-budgets-using-dashboards/
I added costs so far to my ai project tracker (a GitHub repo to track and see if AI can actually build a good real world solution). I created an AWS Cost Management dashboard and convert that to markup and added it to my README. I’ll update it periodically.
I want to see how well my agent wrote the code for my AWS Bootstrap script. One thing I want to check is the infrastructure configuration. Also IAM policies. I don’t want to use AI for that as I’m already burning tokens like crazy.
It’s also going to give me a non-deterministic answer that might miss things. There are a lot of tools you can use to check your infrastructure for adherence to best practices. AWS SecurityHub has a built in CSPM for this.
However, I’m on a budget for this initiatively. If I start making money off what I build on this I’ll spend more.
Be aware if using AWS Config as part of AWS SecurityHub CSPM. Was just using Google to get a gist of the pricing:
—-
Factor In the Hidden AWS Config Dependency
A crucial blind spot for many engineering teams is AWS Config. Security Hub relies completely on AWS Config to track changes in resource configurations. [1]
• AWS Config charges $0.003 per configuration item recorded.
• If you have highly dynamic infrastructure (like ephemeral AWS Lambda functions or auto-scaling container fleets), AWS Config costs can easily scale up and exceed the cost of Security Hub itself.
—-
I believe it is recording every single change all the time. Like every time you deploy and redeploy something. That is a better solution than a static report, but a static one time snapshot report periodically may be easier to handle if you are on a budget.
Anyway I moved all the steps for things I want to run periodically into a separate menu including IAM Access Manager and Prowler. I can add more to that list later if I have time but that’s a start. We’ll see how well the model wrote its code with that.
I’m working towards getting all the deploy steps completed and making progress. I have notes on things I had issues with and how I fixed them in fixed MD file. I can’t write the actual file name because X interprets it as a domain name.
Progress:
I’m still getting 100’s of followers every other day on X and they don’t look real along with a few other top security researchers. I’d like to think it is all students or something good but some of the profiles are so off-topic and most of them have a particular name pattern. Like they took a real name and randomized the last few letter.
But why would someone do this?
Here are some of the ways fake accounts can be used against a profile:
* Suppressing Content: Mass reporting by fake accounts can trigger automated systems to shadowban or suspend targeted users.
* Devaluing Competitors: Competitors may buy fake followers for a rival account to trigger X's spam detection and get the rival suspended.
* Reputation Smears: Networks launch coordinated harassment campaigns to discredit public figures, journalists, or brands.
Hopefully it is none of that but it does not look real.
Just read a great post on quantum computing using the AWS Braket service on the AWS builder center. It got me thinking about the security implications of stuffing multiple qbits or whatever it’s called into the same space where we previously stored 0 or 1.
I’m not using the right terminology here because that was the first time I ever read any details about how quantum computing actually works. I asked the author a few questions about how correctness is determined from what seemed like fuzzy logic. He said it is in fact deterministic but there’s a lot of noise.
I also asked if developers will even know they are using quantum computers in the future or if everything will just be faster. He said the latter.
There are tons of new student programmers and beginner to mid developers on the builder center posting all kinds of things they are working on and learning. But you can also find some really interesting nuggets like that if you are more advanced. Something for everybody!
Got my wheels spinning!
Tracking AI Success Rate On An AWS Infrastructure Project
What are the metrics we really care about when it comes to writing code with AI agents and how can we track them?
Overview
I’ve thought of a lot of different metrics we can use for tracking AI success for the moment I’m going to do something really simple. Here’s what I really care about when it comes to AI Projects - currently 7 things at the time of this writing.
⏰ How much time it takes me to complete
💰How much does it cost
What outputs have been deployed successfully
✅ Has it been tested to prove that it is usable and correct?
🔎 Has a code review been performed to evaluate architectural integrity?
🔒Has it had a security review including code review, assessment and testing?
⏭️ When is my project done so I can move on to a new one?
Read the full story for my initial attempt to track these things on the AWS Bootstrap script I’m working on for Secure AI Agent Infrastructure.
https://teriradichel.substack.com/p/tracking-ai-success-rate-on-an-aws
