It just occurred to me that AI in the nearish future will probably/almost certainly be able to do this.

It was a staple of Asimov’s books that while everyone was improving the robot brain, nobody ever understood how it worked - he said that while the first few generations were programmed by humans, everything since that was programmed by the previous generation of programs.

Obviously this leads us to Asimov’s world in which nobody is even remotely capable of creating programs that violate the assumptions built into the first level of those models - are we at that point now with “AI”?

No. Programs cannot reprogram themselves in a useful way and are very very far from it.

Then how does polymorphic/self-modifying code work?

It doesn’t or do you have serious applications for self-modifying code?