Google is working on an AI agent that can move your mouse, take over your browser

WTF?! There's still a lot of pushback against generative AI from most of the public. Not only are there concerns about the technology taking jobs and plagiarising, but many worry about control being taken out of humans' hands. Google and other companies' solution, it seems, is to create AI agents that can take over your PC, moving the mouse cursor, browsing the web, and entering text.

Google's Project Jarvis will be shown off as soon as December, when it releases the next version of its Gemini LLM, reports The Information. It's the latest in a line of recently announced AI agents that can take limited control of a PC.

Jarvis only works with web browsers – being a Google products means it will be optimized for Chrome. The AI is supposed to automate everyday, web-based tasks by taking screenshots, interpreting the information, then clicking buttons or entering text. The current system takes a few seconds between each action.

Users can also give Jarvis commands directly, such as making purchases, filling out forms, compiling data into tables, opening a series of webpages, or booking flights online.

The idea behind Jarvis is that it will make AI tools more useful and accessible, especially to those with no prior AI experience, as it removes the need to develop APIs. Users just type what they want Jarvis to do and it (should) do it.

Google, a.k.a. Cyberdyne Systems

Google isn't the only company putting more control into the virtual hands of AIs. Anthropic's updated Claude LLM gives users the option of granting the tool limited access and control over a PC. Examples of what Claude can do include filling out forms, planning an outing, and building a website. Anthropic admitted that the system is still "cumbersome and error-prone." Nevertheless, ChatGPT-maker OpenAI is reportedly working on its own version.

Response to the news has been about as negative as one would expect. Beyond the obvious privacy implications, there's also the risk of the AI doing something wrong, which AI systems are prone to do, leading to consequences for users. It seems companies haven't been put off by the outrage over Microsoft Recall, which takes screenshots of everything being done on a PC so users can search through it.

In a totally unrelated piece of news, Google dropped its famous "Don't be evil" motto from its corporate code of conduct just over six years ago.

Permalink to story:

Google is working on an AI agent that can move your mouse, take over your browser

There's a good reason to lock out Google from your computer!

This is so violently anti consumer I won't even touch that part: I trust (some of) the other techspot commenters to thoroughly point that out.

I will point out however, that the people that should be concerned the most are Google's primary customers which are advertisers.

Just think about it despite their promises, this reads as the user having even less input on what they click. It means that Google is basically using less information to construct it's algorithm since well, text input is done by AI, even the most basic choice that dictates consumer trends, the click, is done by AI potentially.

So why exactly you, as an advertiser, are paying Google for? You are paying Google to serve ads to people Google themselves are controlling. Are you actually still reaching the audience you want? Is this actually going to have any input if Google decides what to type and what to click and then serves results and ads accordingly? It seems like a huge scam to me to just charge advertisers to pretend they get great, unprecedented interaction that will slowly but surely translate into just less overall sales for you.

And that's being charitable believe it or not: The alternative is that Google will knowingly select a specific vendor (The one that pays the most gets priority) and intentionally basically create a customer and a sale for you. This means you don't just need advertising, Google basically makes new customers for you out of thin air, which in turn means that you've got NO CHANCE to ever stand on the basis of your product and offer since without Google's help, an AI determines the search term and the click that will lead to a competitor that did paid Google's extortion money to get the sale even if your product was inherently superior, better priced, whatever it might be: without Google you won't sell a single thing.

This means since Google is basically a worst version of Amazon (Since it works for well, the entire internet not just their store front but ALL STORE FRONTS) you're at the whim of whatever they want: changing policies, de-prioritization, restructuring advertising deals, demanding unreasonable terms like a cut of your sales instead of paying upfront for advertising, just whatever they desire.