|
Your weekly dose of Seriously Risky Business news is written by Tom Uren and edited by Patrick Gray and Amberleigh Jack. This week's edition is sponsored by Airlock Digital. You can hear a podcast discussion of this newsletter by searching for "Risky Business News" in your podcatcher or subscribing via this RSS feed.
Listen here The Trump administration has been trying to address two separate AI-related risks in recent months: The specific risk to US national security from China developing powerful AI and the global risk that powerful hacking machines will be available to all and sundry. A proposed bill suggests a sensible way for the US to chip away at both these risks, at least a little. The Collaboration on Adversarial Threats and Security Risks Act proposes safe harbor provisions for AI companies so they can share information related to AI-specific security risks. The idea here is to encourage frontier labs to work together to counter threats from Chinese AI labs, particularly what they describe as IP theft via distillation. Without this bill, sharing this kind of information could fall afoul of anti-trust legislation that prohibits collusion. We know the frontier labs can already detect distillation because they're always complaining about it after the fact. The idea behind this bill is that more permissive information sharing would let them respond quicker and disrupt at least some distillation campaigns. A bill like this isn't necessarily the most effective way for the US government to prevent or deter distillation attacks. But the more direct approaches, like adding Chinese companies to the entities list, come with problems. American companies, for example, are increasingly using open-weight AI models, the best of which are produced by the same Chinese labs that are carrying out large scale distillation campaigns. Indeed, after Treasury Secretary Scott Bessent issued a statement saying that "Entity List designations will be on the table", a tech industry coalition of 76 companies published an open letter supporting the role of open-weight AI models in the technology mix. So, the consensus seems to be that punitive action targeting Chinese labs could be a bonus for the American frontier AI companies, but likely a negative for the US economy writ large. Really, though, the battle for AI dominance is being fought around hardware access, not the models themselves. The US controls chip exports, and there is some evidence that these controls are having an impact. When AI-focussed journalists Lily Ottinger and Kai Williams toured Chinese AI labs in May this year they reported: Basically every AI researcher we talked to in China brought up the same complaint: their companies lack sufficient access to the advanced compute resources necessary to train and run AI models.
American chip export controls are imperfect, but they're biting, and reinforcing them is the most effective way for the US to stay ahead. In the areas around AI that aren't about staying ahead, we think there's room for the United States and China to cooperate. When open-weight models get really good, we suspect neither government will be happy with everyone on the planet having access to powerful AI hacking machines. That time could only be a matter of months away, as the recently released open-weight Chinese Kimi K3 model is roughly equivalent to US frontier models from about six months ago, according to a joint US/UK evaluation. China already has strict AI regulation, but it's focussed on making sure that AI does not upset the Communist Party's information control and censorship apparatus. However, we think Chinese regulators will eventually reign in the release of very capable models. Very capable open-weight models will be used for all sorts of hacking, some of which will make the Chinese Communist Party look bad. The Party will not like that kind of political risk, so cyber safety regulation is coming. But right now there is an opportunity for the US government to engage the Chinese government to encourage and shape this regulation to its advantage. So as an overall strategy, what we've laid out above makes sense: Laws that free up frontier labs to combat distillation collectively; tightening chip controls to make sure the US stays ahead of China; and engaging with Chinese regulators to prevent the spread of top-tier hacking capabilities to all and sundry. Will this be what happens? Long-term strategic thinking is not a Trump admin forte, so we'll hope for the best, but prepare for the status quo. Iran's Perfectly Calibrated Cyber AttackA cyber attack on Minnesota water utilities coupled with a US federal government warning raises the possibility that Iranian hackers have begun targeting US critical infrastructure in earnest. This week more than 30 Minnesota community water systems were targeted by a "coordinated cyberattack", according to the state’s IT services. Disruptions were reported in a number of cities, but the water remained safe to drink. The worst impact reported was in the city of Braham where residents were asked to limit their consumption of water for a few hours as the city was relying on supply in a single water tower. The attack came less than a week after US federal government agencies updated a warning about the possibility of Iranian state-affiliated cyber actors targeting US critical infrastructure. The warning cites incidents since March in which multiple victims across several critical infrastructure sectors had been compromised and that some victims "experienced operational disruption and financial loss". There is no formal attribution of the Minnesota attacks, but security firm Tenable says the "operational pattern" is consistent with the CyberAv3ngers, a threat actor associated with Iran's Islamic Revolutionary Guard Corp (IRGC). In March we wrote that even if the war in Iran successfully achieved the White House's stated objectives, it could still "result in an enduring increase in Iran's capacity and appetite for cyber mayhem". Cyber operations targeting the US and Israel could provide propaganda victories with very little chance of a kinetic response. The Minnesota attacks caused minimal impact, so it would be tempting to argue that they were a flop. Given that the war is still ongoing and also very unpopular, however, we think these attacks might be perfectly calibrated. They're significant enough to generate headlines and public unease, and, maybe, make Americans wonder about the point of the entire war. But they're not lethal or damaging enough to ensure America bands together against a single enemy. Our crystal ball says we can expect more of this. Watch James Wilson and Tom Uren discuss this edition of the newsletter: Three Reasons to Be Cheerful This Week:- US visa restrictions for cybercriminals: Last week, the US Secretary of State Marco Rubio announced visa restrictions for cybercriminals including sextortionists and those running cyber-enabled scams. The policy applies to the " individuals responsible" for the crimes and, in some cases, their family members.
- AI security tools are a thing: This week Microsoft announced a security-focussed AI model that, when coupled with its MDASH harness, is cheaper and better at vulnerability identification and remediation than current frontier models. It's not this specific announcement that makes us cheerful, but rather what it indicates. Microsoft announcing a "world-class security" machine as a product means that there will likely be a number of competitors offering a variety of similar products.
- AI works on cryptographic algorithms too: Anthropic has posted about Claude Mythos Preview being used to find flaws in two different cryptographic systems. One of the results came in a proposed post-quantum algorithm that was being evaluated by NIST. This is a good example of AI being used preventatively to improve standards before they decided.
In this Risky Business sponsor interview, James Wilson chats with Airlock Digital co-founders David Cottingham and Daniel Schell about how attackers are using LLMs to enumerate EDR detections. Listen here
| 
