The Mac Pro still has auxiliary power connectors, so you could theoretically install a GPU in it, just not the highest end ones. No drivers so you can't really do anything with them. I do suppose Nvidia could maybe do CUDA drivers for a card, but would Apple even allow it?
@paul You could theoretically have said the same about thunderbolt eGPU’s on M1 machines. Marcan had posted back on that old social network a couple years ago that the actual problem was related to the base addresses available on the PCIe lanes for hardware and at least in Linux's case too much stuff was hard-coded to rely on how it works elsewhere. (Warning: This is my understanding of what he said)
I had asked a few months ago if that changed for M2 and he had not looked into it yet.

