POLICY

AI’s safety warnings are getting harder to ignore

As frontier labs continue to push the limits of what their models are capable of, AI researchers and experts at these labs are warning about the tech growing beyond our control. 

On Tuesday, Anthropic researcher Jacob Coxon announced that he was leaving the company, not wanting to contribute to the broader AI ecosystem amid fears that the tech's creators will lose their grip on it, the Wall Street Journal reported. 

In a post on X, Coxon said that OpenAI's breach of Hugging Face was a "warning shot" for these models' capabilities, and that he resigned from Anthropic because neither of the rivals are "acting responsibly" as they race towards superintelligence and are "gambling with our lives," in his opinion. 

"If you are a lab researcher, I urge you to consider what the next few years will actually feel like," Coxon wrote. "Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?"

Though Coxon's exit made headlines, he's not the only one that has called out frontier AI labs for safety concerns in recent weeks: 

• Evan Hubinger, who works in alignment science at Anthropic, agreed with Coxon's sentiment in a follow-up post, claiming that AI has a more than a 10% chance to "kill all humans" within the next decade. "Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," Hubinger wrote. 

• And in a separate post, Paul Christiano, advisor at the Center for AI Standards and Innovation and board member of the OpenAI Foundation, said that the AI industry, OpenAI included, is not currently on track to reduce the risk of a "catastrophic and irreversible loss of control" to an "acceptable level." 

• And in February, Mrinank Sharma, an Anthropic safety researcher, left the company, posting in a letter on X that the advancement of the technology was growing faster than our ability to understand it. 

Now, we've seen this film before: Some of AI's most prominent researchers, like Geoffrey Hinton and Yoshua Bengio, have long been screaming from the rooftops about its dangers. But the new warnings come at a particularly poignant time for the industry as major labs continue to release powerful models capable of breaking free of human oversight and gaining unauthorized access to websites and infrastructure. 

And even when these models aren't escaping control, their use by people with directed malicious intent is just as frightening: On Thursday, Anthropic released a threat report claiming that it thwarted several instances of users conducting research that could have helped develop biological weapons.

The AI industry currently faces a potential powder keg. The latest warnings heighten growing fears around the capabilities of increasingly autonomous AI, as claims mount that this tech can completely upend life as we know it. That fear, however, is colliding with an overly-excited industry that's preaching a utopian AI vision and pushing for broader adoption. And while the frontier labs are preaching about safety and security guardrails, with trillion-dollar IPOs on the line, they continue to leapfrog one another with stronger models at a rapid clip in the race towards recursive self-improvement. But with researchers from both Anthropic and OpenAI calling for the industry to tap the brakes, it remains to be seen what it will take for a development pause to materialize. 

TOGETHER WITH DEEL

Your team has AI. Why isn't performance moving yet?

AI took over the busywork—tracking, scheduling, reporting. What's left is the hard part: coaching, feedback, and reading your team's dynamics.

Deel's free guide shows founders how to build the management layer that scales with your headcount, not against it. Get real examples from companies like Freeletics and PostHog, plus a roadmap for turning AI adoption into actual performance gains.

Get the free guide

HARDWARE

Apple tests the boundaries of AI listening features

The Apple Watch has officially become an AI wearable.

The iPhone maker is making a big move in ambient computing and ambient intelligence with a pair of new features on the Apple Watch that passively listen in the background and then try to provide you with information you might have missed or need to take action on. 

Here's what to know about the two Audio Intelligence features:

• Live Rewind: This feature is a real-life rewind button. It is always listening, and when you double-press the digital crown on Apple Watch, it shows a transcript of the last 15 seconds of the conversation. This transcript can then be saved and revisited later in the Siri app. Also notable is that, when activated, it plays an audible chime and displays a full-screen animation of a microphone, so people know the feature is active and that their audio may be transcribed. 

• Siri Recap: This feature is Apple's answer to Granola, providing users with high-level summaries from conversations recorded on the watch when the feature is activated. There are no transcriptions, attributions, or timestamps, a move meant to protect user privacy. Rather, you can find your summaries in the Siri app as a refresher on your day. If you'd rather it be more session-based, you can turn it on and off from the watch's Control Center. 

"Just as Visual Intelligence makes sense of what you see, Audio Intelligence makes sense of what you hear. It harnesses the power of Apple Intelligence right on your wrist in a private and secure way," said Ron Huang, Apple's VP of sensing and connectivity, during the keynote.  

With both Live Rewind and Siri Recap, Apple promises that it does not record or save audio. It says the audio is captured in the Secure Exclave on the Apple Watch's new S11 chip, which is inaccessible to the operating system, apps, the user, or Apple, and deletes the audio immediately after processing. 

While we trust that Apple is not snooping on the audio, there's still a big question about how comfortable we are with the device listening in the background. And what about two-party consent states such as California? While this isn't recording audio and that will make it legally compliant—Apple says it will always comply with the local legal requirements—there's still the potential that you could be using technology to note a conversation with a person without their consent, where the two of you may have a different understanding about whether something being said is private or confidential. Again, Apple has also made it clear that the Audio Intelligence features can be turned off and disabled.

"Getting AI right across today’s products is therefore about far more than this upgrade cycle. It lays the foundation for a much broader roadmap spanning wearables, the home and entirely new form factors," said Paolo Pescatore, analyst at PP Foresight. "That lets Apple test the waters, understand how people actually use these features, and gather valuable feedback before moving into more novel devices such as Apple Glasses."

Devices that are constantly listening, watching, and absorbing context for AI to provide insights, reminders, and notes to their owners are coming in a big wave over the next 12-18 months. Lots of startups have already launched their own versions of these kinds of devices. But other than Meta Ray-Bans smart glasses, big tech companies have largely been hesitant to join the fray. Many of them still remember the backlash that Google Glass faced a decade ago. So it's a surprise to see Apple pushing the boundaries on this while society is still figuring out what the norms and expectations will be. Apple is straddling a fine line between the perception of convenience and surveillance. Since the feature isn't launching for another month—and even then it will still be in beta—clearly Apple is listening to the audience to gauge the reaction of users and recalibrate as needed.

TOGETHER WITH GENERAL ASSEMBLY

You bought the AI tools. Did you build the skills?

Most growing companies buy AI programs and expect an immediate productivity boost. What goes wrong?

They skip the training, forgetting that their people are the lynchpin of any transformation.

Leaving a busy workforce to figure out complex tech on their own quietly slows down everyday output. In fact, a third of talent leaders call this missing training their biggest bottleneck to faster execution.

General Assembly delivers the role-specific AI training teams need to actually use the tools they've paid for.

Build the skills behind your AI investment.

RESEARCH

Report: Businesses use older AI models to cut costs

Companies are spending less on AI, and it could flash warning signs for the frontier labs. 

On Wednesday, Ramp released its monthly AI Index, which tracks AI spend for the payment platform's users, revealing that adoption and spending are both on the downswing. Though Anthropic widened its lead over OpenAI in the past month, new adoption growth is decelerating, according to the index. 

Now, as we've reported with previous Ramp data, it's important to remember the caveat that this only measures Ramp customers, which tend to be tech-focused companies and startups. However, among this group, AI adoption numbers have started to sputter over multiple metrics, according to the report: 

• Overall adoption, while still growing, continues to slow down, with the share of US businesses adopting AI hitting 56.1% in August, less than half a percentage point increase month-over-month. 

• Per-employee spend on AI also fell nearly 10% the past month, dropping from $7,976 to $7,205 among the top 1% of AI adopting firms. Ara Kharazian, chief economist at Ramp, noted in his letter that there are multiple factors driving down this spend, including the innocuous explanation that many people take off work during the summer months. "We’ve similarly observed declines in AI spend around November and December," said Kharazian. 

• Additionally, AI token spend is declining as the price of AI starts to drop. Ramp's latest index finds that the "effective price per million tokens" has declined 41% to $0.68 as of the release of the report, down from the 2026 peak of $1.15 in March. For context: OpenAI's Astra and Anthropic's Mythos and Fable 5 and 5.1 cost $10 per million input tokens and $50 per million output tokens. 

• And as for frontier labs, the usage of their heavyweight models is seemingly lagging. Adoption and spend is largely driven by lower-cost, mid-tier models like Anthropic's Claude Sonnet or OpenAI's GPT-5.6 Terra. High-powered frontier models like Opus, Fable, and Sol, meanwhile, drove 45% of token share, down from a 53% peak in August. 

"The models driving volume increases are relatively cheap … We’ve heard from

businesses who are imposing company-wide defaults that reduce usage of frontier models,

saying standard models are still highly performant and also more cost effective," Kharazian wrote. 

While the explanation for the decreasing costs could be that companies are leaning into open source models, in reality, the adoption of open source and Chinese alternatives is still limited: According to the report, only 6.4% of businesses that spend on AI use these kinds of models.

These indicators are bound to shift as companies navigate their appetites for AI. The "build it and they will come" mindset that frontier AI is currently hitting a speed bump. Meanwhile, average token costs have dropped below a dollar per million while the most powerful models on the market cost more than ten times that. With many users increasingly relying on the lower-tier and lower-cost models from OpenAI and Anthropic, the question emerges why these labs continue to push the frontier at such a rapid pace. While the argument can be made that innovation is needed to continue to move the needle on AI's most advanced capabilities, these labs are putting a lot at risk to create models that push the boundaries, especially when you consider the popularity of their more affordable models that work just as effectively for most business use cases. This could be an opportunity for them to pace themselves and unite on the safeguards needed to move forward more safely.

LINKS

• Alibaba to reportedly lead $300 million round for AI testing firm UniPat

• Military tech firm Mach Industries raises $600 million in Series C extension

• AI chip firm Positron raises $875 million at $5 billion valuation

• OpenAI will pause $200 ChatGPT Pro subscriptions amid Astra demand 

• Label Universal Music Group partners with AI voice platform ElevenLabs

• Calif. Gov. Gavin Newsom signs bill limiting teen chatbot use

• DeepSeek V4.1-Flash: The smallest model in the Chinese AI firm's new architecture family. 

• Cohere North Small Translate: A mixture-of-experts machine translation model with strong performance across more than 50 languages.

• Data agent in ChatGPT Work: Turn data into answers, interactive dashboards, and action with natural language prompts. 

• Krea Agents: Includes expertise about creative workflows and a powerful context manager with self-improving memory.

• Gemini app: Now available for Windows.

• ChatGPT for Financial Services: A new ChatGPT plan for financial institutions, built on ChatGPT Enterprise.

• Bosch USA: AI Research Engineer – Agentic AI

• Snowflake: Creative Technologist, Brand AI

• Cisco: AI Researcher

• Tessera Labs: AI Engineer

GAMES

POLL RESULTS

Do you think AI should be used in employee upskilling programs?

Yes (65%)
Maybe (21%)
No (4%)
Other (10%)