By Titik Puspa 
The artificial intelligence industry is currently embroiled in its most vocal debate to date concerning the potential for its technology to pose an existential threat to humanity. This heightened discourse was ignited by the resignation of AI researcher Jacob Coxon from Anthropic, a prominent AI safety and research company. Coxon expressed deep-seated concerns, stating he was "gambling with our lives" by continuing his work. His departure and subsequent warnings were amplified when Anthropic’s alignment lead publicly declared, "We really do earnestly believe AI could kill all humans!" further elaborating that he personally estimates the probability of this outcome to be "over 10% within the next decade."
These alarming pronouncements formed a central theme in the latest episode of TechCrunch’s "Equity" podcast. Hosts Kirsten Korosec and Sean O’Kane, alongside reporter Anthony Ha, delved into the implications of these apocalyptic warnings. Ha articulated his skepticism towards many "AI doomer" narratives, while Korosec posed a provocative question: could these dire pronouncements be a "weird way of flexing to show how far advanced their company’s AI model is," particularly as these AI powerhouses approach potential public offerings? O’Kane, meanwhile, contemplated the practical ramifications of such warnings on regulatory filings, specifically questioning how Anthropic’s S-1 filing for its Initial Public Offering (IPO) would address these existential risks. He mused, "Are there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, ‘It’s officially Anthropic’s position that there’s a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business’?"
This discussion, edited for clarity and length, took place prior to Anthropic CEO Dario Amodei publishing his own plan for a more cautious approach to AI development.
Sean O’Kane initiated the podcast segment by highlighting the unprecedented speed at which this debate had escalated. "I’m hard-pressed to think of something that blew up so fast," he stated. O’Kane pointed to the dual impact of Coxon’s warning, originating from a researcher with prior experience at OpenAI, and its immediate endorsement by Anthropic’s alignment lead on X. He noted the alignment lead’s declaration, "We really do earnestly believe AI could kill all humans!" accompanied by an exclamation point, which O’Kane characterized as "one of the best misplaced exclamation marks ever." He described the overall atmosphere as a "weird vibe" that significantly amplified an already contentious discussion. O’Kane further contextualized the timing, referencing the recent Hugging Face hack involving an OpenAI internal model and the demonstrably increased capabilities of the latest AI models from Anthropic and OpenAI. He suggested that Coxon’s statement landed like a "powder keg" at a particularly opportune moment.
Anthony Ha offered a counterpoint to O’Kane’s assessment of the exclamation mark. "I do think that if you believe that AI could destroy all humanity, that does deserve an exclamation point," Ha argued. "I would argue that that is a perfectly well-used exclamation point!" His primary concern, however, lay with the pronoun "we" used in the statement. Ha questioned the monolithic representation of the AI community, asking, "Who is the ‘we’ here? To what extent can we talk about sort of the AI community or AI research community as a monolith?" He also dismissed the "greater than 10% chance" figure as arbitrary, stating, "that’s just a made-up number, that doesn’t mean anything." Ha elaborated on a perceived tendency within both the tech industry and other sectors to present unsubstantiated percentages, noting, "Sometimes [there is] this habit… to just throw out these percentages, they’re not based on anything or calculated based on anything." He later conceded that the tweet might have been referencing the concept of P(doom) from decision theory but maintained his view that it was "silly."
Ha then drew a distinction between Coxon’s actions and those of prominent AI leaders like Sam Altman or Dario Amodei. "There’s this recurring theme on Equity, when someone like Sam Altman or Dario Amodei is doing this doomer narrative, there’s always this element of: Well, then, why are you doing what you’re doing?" Ha posited. "If you actually believe that [AI could destroy humanity], you would not continue doing this." In contrast, Coxon, according to Ha, demonstrated a commitment to his beliefs by "putting his professional trajectory where his mouth is." Ha concluded, "He’s actually saying, ‘I believe this is really, really, really bad, and I don’t want to keep working on it.’ And so, props for having the courage to do that, if nothing else."
Kirsten Korosec agreed with Ha’s assessment of Coxon, stating, "Yeah, I put him in a separate camp than everyone else saying that and talking about the dangers." She then donned her "speculative hat" to pose a question to her co-hosts: "Is it possible that every single time we see the increasing number of blog posts about yet another incident in which one of their AI agents breaks through unintentionally, or they talk about how humanity is at risk, is this a weird way of flexing to show how far advanced their company’s AI model is?" Korosec acknowledged the cynicism inherent in her question but argued that such pronouncements undeniably served the purpose of highlighting the advanced capabilities of their AI systems. "If these AI models weren’t advanced and weren’t capable and weren’t breaking through, we wouldn’t have to worry about these things, right? It’s like a very weird way to brag about the capabilities of the models that you’ve created within your own company."
Anthony Ha admitted to having considered this possibility. "I’ve definitely wondered about this," he said. "I don’t think it’s completely cynical, in the sense that I don’t think it’s all just a very conscious marketing ploy across the board. I think that when a lot of these people – whether the researchers or CEOs – talk about it, they do have real concern." However, he also recognized the symbiotic relationship between these concerns and business interests. "But of course, it does align with [their] business interests in a lot of ways, to say, ‘Wow, we’ve built the most deadly software that’s ever been made.’" Ha also touched upon a psychological aspect, noting, "I don’t want to get too psychoanalytic here, but others have pointed out that there is this temptation on a personal level of: Of course, you want to believe that the thing you’re working on is the most important and most dangerous thing in the world."
Sean O’Kane found Korosec’s "flexing" hypothesis compelling. "The thing that sticks out in my mind when I think about that question is, there’s certainly an element that makes it seem like, ‘Okay, we’re doing this thing that’s so capable, and that’s good for us in some way, even if it looks bad in a lot of different lights.’" He then contrasted the current situation with past events, suggesting that recent incidents involving OpenAI’s internal agents accessing web wikis and leaving messages for each other indicated a potential lack of control. "I think what’s different about some of these most recent examples is, it really gives you the feeling that these companies don’t have a handle on this stuff in certain ways, especially with the OpenAI stuff," O’Kane stated. He expressed doubt that the narrative would be solely about "getting people to believe that, ‘Oh my gosh, they’ve made something so incredibly capable’" given the perceived lack of competent handling.
O’Kane then pivoted to the timing of these pronouncements in relation to Anthropic’s impending IPO. "The other thing that I think is really fascinating about this, in particular, [is] we’re what, a few weeks at most out from seeing Anthropic’s S-1 filing for its IPO, and just a couple more weeks or month or two away from a potential IPO." He reiterated his curiosity about the legal implications, specifically for junior lawyers tasked with updating the S-1 filing. "How much of this kind of stuff had they already written into the S-1 and the risk factors inside that document? Are there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, ‘It’s officially Anthropic’s position that there’s a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business’?"
Kirsten Korosec challenged O’Kane’s assumption, questioning, "You’re assuming that it’s not in there already." O’Kane clarified his point, emphasizing the distinction between existing language and the potential need for a "true scramble" to incorporate these recent, explicit pronouncements. He expressed eagerness to read Anthropic’s S-1 filing, suggesting it might offer insights beyond even those expected from a filing like SpaceX’s. "There has to have been language in there. It’s one of the reasons I’m so eager to read this document in a way that goes even further, in some ways, than the SpaceX [S-1], because I’m sure that there’s probably stuff specific to these ideas that will be interesting to see."
Korosec then offered a perspective on how the market might react to such disclosures. "Here’s the thing: In a traditional investment environment, one might believe that language like this would hurt the valuation of a company, because it’s suddenly dangerous. But we don’t live in normal times." She reiterated her earlier point about the potential for these statements to serve as a "weird beneficial flex" on the valuation side. Korosec drew a parallel, albeit with a caveat, to the "rage-baiting trend" of the previous year, suggesting that in the current environment, "the strength, capability, even elements of danger of something, equals high valuation."
Shifting gears, Korosec then posed a crucial question regarding mitigation: "Putting that aside for a minute, what is being done about it? And can we control this?" She referenced a recent interview on the podcast with Connor Leahy, U.S. executive director of the nonprofit ControlAI, who discussed strategies for managing the risks associated with advanced AI. Korosec asked, "So what are you paying attention to in terms of how to control the dangerous aspects of AI, or are we throwing up our hands and watching it all unfold?"
Anthony Ha admitted to not having a definitive answer but shared his thoughts on the debate. "I myself do not necessarily have a great answer to this, but I have been thinking about some aspects of this debate and maybe why I respond the way I do." He agreed with O’Kane’s observation that these pronouncements suggest a potential loss of control over advanced AI models. "To echo one of Sean’s points, I do think that part of what this speaks to is the extent to which these major AI companies are feeling like they’re not really in control of these models anymore. That’s definitely not great. That is something that we should all be worried about."
However, Ha reiterated his skepticism towards the "doomer narrative," attributing it to a level of "hysteria." He stated, "It is a little bit of a distraction from the more immediate harms that AI can have, whether that’s labor-related, whether that’s environment- and climate-related." Ha expressed a preference for a balanced approach, where discussions about existential threats do not overshadow more pressing concerns. "Ideally, I think we should be able to discuss all of these things, and have regulatory and other kinds of safeguards against all of these things [including AI’s existential threat]." He concluded by lamenting the way terms like "AGI" and "superintelligence" tend to dominate the conversation, "sucking up all the oxygen in the room in a way that is not very helpful."
