The Mac Pro still has auxiliary power connectors, so you could theoretically install a GPU in it, just not the highest end ones. No drivers so you can't really do anything with them. I do suppose Nvidia could maybe do CUDA drivers for a card, but would Apple even allow it?
@paul WTH, How many watts do GPUs take now?? I haven't bought a separate GPU for nearly 20 years…
@pixel @paul My new 4090 came with a power plant.
@steveriggins Holy crap. some f the 4090s need 1200 watts. TWELVE HUNDRED. https://www.tomshardware.com/news/1200w-power-requirement-rtx-4090
Some RTX 4090 Graphics Cards Recommend a 1200W Power Supply

That might be overkill, even for an RTX 4090

Tom's Hardware
@pixel well, there goes my fantasy of a new MacPro loaded with extra A100s.

@Dhmspector @pixel My understanding is that the new Mac Pro doesn’t support discrete GPUs.

I’d love to be wrong about this, though.

@jeff been the case for a long while… but I can dream.