The Mac Pro still has auxiliary power connectors, so you could theoretically install a GPU in it, just not the highest end ones. No drivers so you can't really do anything with them. I do suppose Nvidia could maybe do CUDA drivers for a card, but would Apple even allow it?
@paul WTH, How many watts do GPUs take now?? I haven't bought a separate GPU for nearly 20 years…
@pixel @paul My new 4090 came with a power plant.

Some RTX 4090 Graphics Cards Recommend a 1200W Power Supply
That might be overkill, even for an RTX 4090
Tom's Hardware@pixel well, there goes my fantasy of a new MacPro loaded with extra A100s.