On the large AI models, this preprint synthesises what we know so far https://arxiv.org/abs/2209.15259
In short: it is mathematically impossible to have AIs combining the following properties:
1) High number of parameters
2) Robustness to poisoning (e.g. fake data)
3) Privacy-preserving
On the Impossible Safety of Large AI Models
Large AI Models (LAIMs), of which large language models are the most prominent recent example, showcase some impressive performance. However they have been empirically found to pose serious security issues. This paper systematizes our knowledge about the fundamental impossibility of building arbitrarily accurate and secure machine learning models. More precisely, we identify key challenging features of many of today's machine learning settings. Namely, high accuracy seems to require memorizing large training datasets, which are often user-generated and highly heterogeneous, with both sensitive information and fake users. We then survey statistical lower bounds that, we argue, constitute a compelling case against the possibility of designing high-accuracy LAIMs with strong security guarantees.
