Will AI End Humanity? Exploring the Existential Risks and Ethics of Superintelligent System

Artificial Intelligence (AI) presents humanity with a profound dilemma: the promise of revolutionary advancements, such as accelerating economic growth and optimizing healthcare, balanced against the potential for catastrophic or existential risk. Experts project that Artificial General Intelligence (AGI) may reach human-level intelligence within approximately the next two decades, with many expecting it to surpass human intelligence rapidly thereafter, posing a significant existential threat.

We explore the two primary pathways by which AI could cause existential catastrophes:

1. Decisive AI X-Risk: The conventional view, which envisions an abrupt, cataclysmic event caused by a highly advanced AI, typically Artificial Superintelligence (ASI). This pathway is characterized by a single, overwhelming impact, such as scenarios where a misaligned ASI pursues instrumental subgoals (like resource acquisition or self-preservation) that inadvertently lead to human annihilation.
2. Accumulative AI X-Risk: An alternative pathway suggesting that AI x-risks emerge gradually through the compounding impact of multiple smaller, interconnected AI-induced disruptions over time. This perspective is likened to a "boiling frog" scenario, where incremental AI risks slowly erode systemic and societal resilience until a modest perturbation triggers an unrecoverable collapse.

These accumulating risks stem from a variety of near-term ethical and social concerns. We examine concrete risk factors that can evolve into existential threats, including:

• Misalignment and Inequity: Failing to align AI with human values can perpetuate existing inequities and cause real-world harm, such as biased diagnostic tools or algorithms inadvertently prioritizing certain patient groups in healthcare.
• Overtrust and Misinformation: Blind reliance on AI for critical tasks like medical diagnosis could lead to catastrophic errors. Furthermore, advanced AI systems can generate convincing misinformation with high confidence, undermining public trust and potentially destabilizing societal structures, especially during crises.
• Privacy and Security: Sophisticated AI systems are capable of memorizing and reproducing personally identifiable information, raising serious privacy concerns. Malicious actors could weaponize these privacy risks for large-scale surveillance or targeted exploitation, potentially enabling bioterrorism.
• Economic and Societal Destabilization: The concentration of large foundation models among a few corporations creates a risk of monopolization and manipulation. Furthermore, AI-enabled automation poses threats of economic displacement, exacerbating disparities and restricting opportunities for disempowered communities.

From an economic perspective, the core AI Dilemma involves calculating the optimal use of AI, weighing the massive consumption gains it could provide (potentially leading to a technological "singularity") against the risk of extinction. The degree of existential risk society is willing to tolerate depends significantly on the assumed curvature of utility. However, one key insight suggests that if AI innovations are capable of extending life expectancy, the existential risk cutoffs can be much higher, making large existential risks more bearable because mortality improvements and existential risk are, loosely, "in the same units".

Finally, we consider the intense debate surrounding AI risk prioritization. Critics argue that focusing extensively on speculative future doomsday scenarios is a Distraction that diverts attention and resources away from regulating and addressing real, immediate harms caused by current AI systems, such as bias, misinformation, and privacy violations.