Somewhat unpopular opinion: Lightning was by far the better connector.

So why does Apple use USB-C on iPad Pro and MacBooks?

Because they co-developed it and it is the better connector if you want to do anything other than charging.

If they co-developed it, why did it take the EU forcing them to switch to it?

Because they wanted to stick with their lightning ecosystem for iPhone

Read: they wanted to make money by being a huge pain in the dick