"Behind the scenes, Google’s agent is taking screenshots of your browser window, something users must agree to in the terms of service, and sending them to Gemini in the cloud for processing. Gemini then sends instructions back to your computer to navigate the web page."
https://techcrunch.com/2024/12/11/google-unveils-project-mariner-ai-agents-to-use-the-web-for-you/