I know of a few security companies that use slack to work together that includes a shitton of privat data, source codes and confidentional information

Guess whoever introduced the company to slack service fucked up by not reading their policies.

I’m working in fintech, and we share pii through DMs all the time (for investigation purposes). I’d be really surprised if the AI would need to train on that.