When I was taught how to interview for pentesters (I was a project manager at the time so I would ask soft-skill questions). One of the things we looked for was someone who could acknowledge that they didn’t know something.
We would ask the interviewers to review a candidates resume and find a technical gap. One that the interviewer had more technical depth in. They didn’t even need to attempt to answer, just be willing to acknowledge they didn’t know something technical. We knew that making something up or unfounded confidence was the death of consultancy reputations.
It filtered out a massive amount of people.
If you ask a person for their best guess and they say, “ I cannot do that it would be unethical or dangerous to speculate. I don’t know.” those are the people I argue you want to surround yourself with. Especially now.
Now the marketed “expert” AI tries answer everything and anything. Even if it should not. By proxy so too does the AI operator who will bubble that up chain. There’s not even an acknowledgement of the data sets touched or used in the inference stage to give replies. Worst yet, the chatbot largely serves a function inside of business contexts of being the scapegoats for the shameless employee or managers.
The last leg of judgment “Am I qualified to answer?” and “Do I know what I don’t know?” is ignored. I’m curious of the downstream effects that truly has. It scares me.
I’m worried because I’m seeing autonomous pentesting, AI E2E pentesting, and Expert AI security agents. They will never tell you “No” or “I don’t know”. We already had a problem with security consultancies misrepresenting their work and performing it despite not being capable of performing it properly.
To those who view pentesting as a way to serve the people downstream of the companies they advise, it’s a slap in the face. The founders of these AI companies will be rewarded for never understanding or learning. They will show the money they make as evidence of success and market fit. Worse yet, we know that these datasets largely come from the training labs hawked at juniors to upskill.
I’m think I’m also jaded about the pentesting profession. I dropped out of junior year of uni to work at a pentesting firm when I was 21. I wanted to continue studying but my private student loans @ 15% interest continuing to tick up terrified me. My options were to either join the U.S. military, roll the dice with compound interest, or work in the field I wanted to work in since being 10 and finding out CheatEngine could get me max KinzCash on WebKinz. I made my choice.
We wonder why wealthy or spoiled individuals lose touch with reality or at least the reality of the average person. When viewing wealthy reality through the lens of a prolonged exposure to “Yes Men” damages the psychology of the individual, it starts to feel eerily similar to those who are overusing these LLM tools.
Being told “Yes” is addicting at a deep brain chemistry level. Power is also addicting, it inflates the ego. It causes an incremental normalization leading to delusions of grandeur. When I talk to people who use LLMs excessively, it reminds me of the C-levels I’ve seen wreck organizations. It reminds me of the middle managers I worked with. Well, more like the ones who took unearned credit for the work they never understood.
You are not immune to social engineering, addiction, or brain chemistry just because you can intellectualize about it. I think it’s a mistake that companies are trying to mandate LLM usage. We should view as if we were forced to participate in company smoke breaks.
I worry because I think this about my own profession.
I worry because I don’t know enough about other fields to know all the ways I should be scared.
I worry because a lot of us are seeing a mirror being held up to humanity when given a Yes-Man to coax them.
When I speculate, the conclusion I come to is we are in a massive global trust loss event. I think with that eventually comes a great decentralization. I don’t really know what individuals can do.
So instead: I write my thoughts, I make dumb art, and I try to go outside.
Please enjoy a picture of a mouse puppet reading a book. I took the picture myself from inside a castle.
