The agents are arriving
When I first wrote about the risks of powerful AI systems displacing or destroying humanity, one very reasonable question was this: How could an AI act against humanity, when they really don’t act at all?
This reasoning is right, as far as current technology goes. Claude or ChatGPT, which just respond to user prompts and don’t act independently in the world, can’t execute on a long-term plan; everything they do is in response to a prompt, and almost all that action takes place within the chat window.
But AI was never going to remain as a purely responsive tool simply because there is so much potential for profit in agents. People have been trying for years to create AIs that are built out of language models, but which make decisions independently, so that people can relate to them more like an employee or an assistant than like a chatbot.
Generally, this works by creating a small internal hierarchy of language models, like a little AI company. One of the models is carefully prompted and in some cases fine-tuned to do large-scale planning. It comes up with a long-term plan, which it delegates to other language models. Various sub-agents check their results and change approaches when one sub-agent fails or reports problems.
The concept is simple, and Manus is far from the first to try it. You may remember that last year we had Devin, which was marketed as a junior software engineering employee. It was an AI agent that you interacted with via Slack to give tasks, and which it would then work on achieving without further human input except, ideally, of the kind a human employee might occasionally need.
The economic incentives to build something like Manus or Devin are overwhelming. Tech companies pay junior software engineers as much as $100,000 a year or more. An AI that could actually provide that value would be stunningly profitable. Travel agents, curriculum developers, personal assistants — these are all fairly well-paid jobs, and an AI agent could in principle be able to do the work at a fraction of the cost, without needing breaks, benefits, or vacations.
But Devin turned out to be overhyped, and didn’t work well enough for the market it was aiming at. It’s too soon to say whether Manus represents enough of an advance to have real commercial staying power, or whether, like Devin, its reach will exceed its grasp.
I’ll say that it appears Manus works better than anything that has come before. But just working better isn’t enough — to trust an AI to spend your money or plan your vacation, you’ll need extremely high reliability. As long as Manus remains tightly limited in availability, it’s hard to say if it will be able to offer that. My best guess is that AI agents that seamlessly work are still a year or two away — but only a year or two.
The China angle
Manus isn’t just the latest and greatest attempt at an AI agent.
It is also the product of a Chinese company, and much of the coverage has dwelled on the Chinese angle. Manus is clearly proof that Chinese companies aren't just imitating what’s being built here in America, as they’ve often been accused of doing, but improving on it.
That conclusion shouldn’t be shocking to anyone who is aware of China’s intense interest in AI. It also raises questions about whether we will be thoughtful about exporting all of our personal and financial data to Chinese companies that are not meaningfully accountable to US regulators or US law.
Installing Manus on your computer gives it a lot of access to your computer — it’s hard for me to figure out the exact limits on its access or the security of its sandbox when I can’t install it myself.
One thing we’ve learned in digital privacy debates is that a lot of people will do this without thinking about the implications if they feel Manus offers them enough convenience. And as the TikTok fight made clear, once millions of Americans love an app, the government will face a steep uphill battle in trying to restrict it or oblige it to follow data privacy rules.
But there are also clear reasons Manus came out of a Chinese company and not out of, say, Meta — and they’re the very reasons we might prefer to use AI agents from Meta. Meta is subject to US liability law. If its agent makes a mistake and spends all your money on website hosting, or if it steals your Bitcoin or uploads your private photos, Meta will probably be liable. For all of these reasons, Meta (and its US competitors) are being cautious in this realm.
I think caution is appropriate, even as it may be insufficient. Building agents that act independently on the internet is a big deal, one that poses major safety questions, and I’d like us to have a robust legal framework about what they can do and who is ultimately accountable.
But the worst of all possible worlds is a state of uncertainty that punishes caution and encourages everyone to run agents that have no accountability at all. We have a year or two to figure out how to do better. Let’s hope Manus prompts us to get to work on not just building those agents, but building the legal framework that will keep them safe.
—Kelsey Piper, senior writer
