|
|
| (Eine dazwischenliegende Version desselben Benutzers wird nicht angezeigt) |
| Zeile 1: |
Zeile 1: |
| {{Short description|Hypothetical agent surpassing human intelligence}}
| | A '''superintelligence''' is a hypothetical agent that possesses intelligence surpassing that of the most gifted human minds.<sup>[1]</sup> Philosopher Nick Bostrom defines ''superintelligence'' as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest".<sup>[2]</sup> |
| {{for multi|the book by Nick Bostrom|Superintelligence: Paths, Dangers, Strategies|the 2020 film|Superintelligence (film)}}
| |
|
| |
|
| A '''superintelligence''' is a hypothetical [[intelligent agent|agent]] that possesses [[intelligence]] surpassing that of the brightest and most [[intellectual giftedness|gifted]] human minds.<ref>{{Cite web |last1=Mucci |first1=Tim |last2=Stryker |first2=Cole |date=2023-12-14 |title=What Is Artificial Superintelligence? {{!}} IBM |url=https://www.ibm.com/think/topics/artificial-superintelligence |access-date=2025-04-17 |website=www.ibm.com |language=en}}</ref> "Superintelligence" may also refer to a property of advanced problem-solving systems that [[narrow AI|excel in specific areas]] (e.g., superintelligent [[Neural machine translation|language translators]] or engineering assistants). Nevertheless, a general purpose superintelligence remains hypothetical and its creation may or may not be triggered by an [[Technological singularity#Intelligence explosion|intelligence explosion]] or a [[technological singularity]].
| | Technological researchers disagree about how likely present-day human intelligence is to be surpassed. Some argue that advances in artificial intelligence (AI) will probably result in general reasoning systems that lack human cognitive limitations. Others believe that humans will evolve or directly modify their biology to achieve radically greater intelligence.<sup>[3][4]</sup> Several future study scenarios combine elements from both of these possibilities, suggesting that humans are likely to interface with computers, or upload their minds to computers, in a way that enables substantial intelligence amplification. The hypothetical creation of the first superintelligence may or may not result from an intelligence explosion or a technological singularity. |
|
| |
|
| [[University of Oxford]] philosopher [[Nick Bostrom]] defines ''superintelligence'' as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest".{{sfn|Bostrom|2014|loc=Chapter 2}} The program [[Fritz (chess)|Fritz]] falls short of this conception of superintelligence—even though it is much better than humans at chess—because Fritz cannot outperform humans in other tasks.{{sfn|Bostrom|2014|p=22}}
| | Some researchers believe that superintelligence will likely follow shortly after the development of artificial general intelligence. The first generally intelligent machines are likely to immediately hold an enormous advantage in at least some forms of mental capability, including the capacity of perfect recall, a vastly superior knowledge base, and the ability to multitask in ways not possible to biological entities. |
|
| |
|
| Technological researchers disagree about how likely present-day [[human intelligence]] is to be surpassed. Some argue that advances in [[artificial intelligence]] (AI) will probably result in general reasoning systems that lack human cognitive limitations. Others believe that humans will evolve or directly modify their biology to achieve radically greater intelligence.<ref>{{Citation|last=Pearce|first=David|title=The Biointelligence Explosion: How Recursively Self-Improving Organic Robots will Modify their Own Source Code and Bootstrap Our Way to Full-Spectrum Superintelligence |date=2012|url=http://link.springer.com/10.1007/978-3-642-32560-1_11|work=Singularity Hypotheses|series=The Frontiers Collection |pages=199–238|editor-last=Eden|editor-first=Amnon H.|place=Berlin, Heidelberg|publisher=Springer Berlin Heidelberg|doi=10.1007/978-3-642-32560-1_11|isbn=978-3-642-32559-5|access-date=2022-01-16|editor2-last=Moor|editor2-first=James H.|editor3-last=Søraker|editor3-first=Johnny H.|editor4-last=Steinhart|editor4-first=Eric|url-access=subscription}}</ref><ref>{{Cite book|title=The Age of Artificial Intelligence: An Exploration|year=2020|isbn=978-1-62273-872-4|editor-last=Gouveia|editor-first=Steven S.|chapter=ch. 4, "Humans and Intelligent Machines: Co-evolution, Fusion or Replacement?", David Pearce|publisher=Vernon Press |chapter-url=https://www.biointelligence-explosion.com/parable.html}}</ref> Several [[futures studies|future study]] scenarios combine elements from both of these possibilities, suggesting that humans are likely to [[brain–computer interface|interface with computers]], or [[mind uploading|upload their minds to computers]], in a way that enables substantial [[intelligence amplification]].
| | Several scientists and forecasters have been arguing for prioritizing early research into the possible benefits and risks of human and machine cognitive enhancement, because of the potential social impact of such technologies.<sup>[5]</sup> |
|
| |
|
| Some researchers believe that superintelligence will likely follow shortly after the development of [[artificial general intelligence]]. The first generally intelligent machines are likely to immediately hold an enormous advantage in at least some forms of mental capability, including the capacity of [[Eidetic memory|perfect recall]], a vastly superior knowledge base, and the ability to [[Human multitasking|multitask]] in ways not possible to biological entities. This may allow them to — either as a single being or as a new [[species]] — become much more powerful than humans, and displace them.{{sfn|Bostrom|2014|loc=Chapter 2}}
| | == Artificial superintelligence == |
| | Philosopher David Chalmers argues that artificial general intelligence is a very likely path to '''artificial superintelligence''' ('''ASI'''). Chalmers breaks this claim down into an argument that AI can achieve ''equivalence'' to human intelligence, that it can be ''extended'' to surpass human intelligence, and that it can be further ''amplified'' to completely dominate humans across arbitrary tasks.<sup>[6]</sup> |
|
| |
|
| Several scientists and forecasters have been arguing for prioritizing early research into the possible benefits and risks of [[Intelligence amplification|human and machine cognitive enhancement]], because of the potential [[Social influence|social impact]] of such technologies.{{sfn|Legg|2008|pp=135–137}}
| | Concerning human-level equivalence, Chalmers argues that the human brain is a mechanical system, and therefore ought to be emulatable by synthetic materials.<sup>[7]</sup> He also notes that human intelligence was able to biologically evolve, making it more likely that human engineers will be able to recapitulate this invention. Evolutionary algorithms, in particular, should be able to produce human-level AI.<sup>[8]</sup> Concerning intelligence extension and amplification, Chalmers argues that new AI technologies can generally be improved on, and that this is particularly likely when the invention can assist in designing new technologies.<sup>[9]</sup> |
|
| |
|
| == Feasibility of artificial superintelligence ==
| | An AI system capable of self-improvement could enhance its own intelligence, thereby becoming more efficient at improving itself. This cycle of "recursive self-improvement" might cause an intelligence explosion, resulting in the creation of a superintelligence.<sup>[10]</sup> |
| [[File:Test scores of AI systems on various capabilities relative to human performance - Our World in Data.png|thumb|upright=1.7|Artificial intelligence, especially [[foundation model]]s, has made rapid progress, surpassing human capabilities in various [[Benchmarks for artificial intelligence|benchmarks]].]]
| |
|
| |
|
| The creation of '''artificial superintelligence''' ('''ASI''') has been a topic of increasing discussion in recent years, particularly with the rapid advancements in [[artificial intelligence]] (AI) technologies.<ref>{{Cite news |date=2025-01-06 |title='Superintelligence' is the next big thing for OpenAI: Sam Altman |url=https://economictimes.indiatimes.com/tech/artificial-intelligence/superintelligence-is-the-next-big-thing-for-openai-sam-altman/articleshow/116993208.cms?from=mdr |access-date=2025-02-01 |work=The Economic Times |issn=0013-0389}}</ref><ref>{{Cite web |date=2024-06-20 |title=OpenAI co-founder Sutskever sets up new AI company devoted to 'safe superintelligence' |url=https://apnews.com/article/openai-sutskever-altman-artificial-intelligence-safety-c6b48a3675fb3fb459859dece2b45499 |access-date=2025-02-01 |website=AP News |language=en}}</ref>
| | Computer components already greatly surpass human performance in speed. Bostrom writes, "Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~2 GHz)."<sup>[11]</sup> Moreover, neurons transmit spike signals across axons at no greater than 120 m/s, "whereas existing electronic processing cores can communicate optically at the speed of light". Thus, the simplest example of a superintelligence may be an emulated human mind running on much faster hardware than the brain. A human-like reasoner who could think millions of times faster than current humans would have a dominant advantage in most reasoning tasks, particularly ones that require haste or long strings of actions. |
|
| |
|
| === Progress in AI and claims of AGI ===
| | Another advantage of computers is modularity, that is, their size or computational capacity can be increased. A non-human (or modified human) brain could become much larger than a present-day human brain, like many supercomputers. Bostrom also raises the possibility of ''collective superintelligence'': a large enough number of separate reasoning systems, if they communicated and coordinated well enough, could act in aggregate with far greater capabilities than any sub-agent. |
|
| |
|
| Recent developments in AI, particularly in [[large language model]]s (LLMs) based on the [[Transformer (machine learning model)|transformer]] architecture, have led to significant improvements in various tasks. Models like [[GPT-3]], [[GPT-4]], [[Claude 3.5]] and others have demonstrated capabilities that some researchers argue approach or even exhibit aspects of [[artificial general intelligence]] (AGI).<ref>{{Cite web |title=Microsoft Researchers Claim GPT-4 Is Showing "Sparks" of AGI |url=https://futurism.com/gpt-4-sparks-of-agi |access-date=2023-12-13 |website=Futurism |date=23 March 2023 }}</ref>
| | Humans outperform non-human animals in large part because of new or enhanced reasoning capacities, such as long-term planning and language use. (See evolution of human intelligence and primate cognition.) If there are other possible improvements to reasoning that would have a similarly large impact, this makes it more likely that an agent can be built that outperforms humans in the same fashion humans outperform chimpanzees.<sup>[12]</sup> |
|
| |
|
| However, the claim that current LLMs constitute AGI is controversial. Critics argue that these models, while impressive, still lack true understanding and are primarily sophisticated pattern matching systems.<ref>{{Cite arXiv |last1=Marcus |first1=Gary |last2=Davis |first2=Ernest |title=GPT-4 and Beyond: The Future of Artificial Intelligence |eprint=2303.10130 |year=2023|class=econ.GN }}</ref>
| | The above advantages hold for artificial superintelligence, but it is not clear how many hold for biological superintelligence. Physiological constraints limit the speed and size of biological brains in many ways that are inapplicable to machine intelligence. As such, writers on superintelligence have devoted much more attention to superintelligent AI scenarios.<sup>[13]</sup> |
|
| |
|
| === Pathways to superintelligence === | | === Projects === |
| | In 2024, Ilya Sutskever left OpenAI to cofound the startup Safe Superintelligence, which focuses solely on creating a superintelligence that is safe by design, while avoiding "distraction by management overhead or product cycles".<sup>[14]</sup> Despite still offering no product, the startup became valued at $30 billion in February 2025.<sup>[15]</sup> |
|
| |
|
| Philosopher [[David Chalmers]] argues that AGI is a likely path to ASI. He posits that AI can achieve equivalence to [[human intelligence]], be extended to surpass it, and then be amplified to dominate humans across arbitrary tasks.{{sfn|Chalmers|2010|p=7}}
| | In 2025, Meta created Meta Superintelligence Labs, a new AI division led by Alexandr Wang.<sup>[16]</sup> |
|
| |
|
| More recent research has explored various potential pathways to superintelligence:
| | == Biological superintelligence == |
| | Carl Sagan suggested that the advent of Caesarean sections and ''in vitro'' fertilization may permit humans to evolve larger heads, resulting in improvements via natural selection in the heritable component of human intelligence.<sup>[17]</sup> By contrast, Gerald Crabtree has argued that decreased selection pressure is resulting in a slow, centuries-long reduction in human intelligence and that this process instead is likely to continue. There is no scientific consensus concerning either possibility and in both cases, the biological change would be slow, especially relative to rates of cultural change. |
|
| |
|
| # Scaling current AI systems – Some researchers argue that continued scaling of existing AI architectures, particularly transformer-based models, could lead to AGI and potentially ASI.<ref>{{Cite arXiv |last1=Kaplan |first1=Jared |last2=McCandlish |first2=Sam |last3=Henighan |first3=Tom |last4=Brown |first4=Tom B. |last5=Chess |first5=Benjamin |last6=Child |first6=Rewon |last7=Gray |first7=Scott |last8=Radford |first8=Alec |last9=Wu |first9=Jeffrey |last10=Amodei |first10=Dario |title=Scaling Laws for Neural Language Models |year=2020|class=cs.LG |eprint=2001.08361 }}</ref>
| | Selective breeding, nootropics, epigenetic modulation, and genetic engineering could improve human intelligence more rapidly. Bostrom writes that if we come to understand the genetic component of intelligence, pre-implantation genetic diagnosis could be used to select for embryos with as much as 4 points of IQ gain (if one embryo is selected out of two), or with larger gains (e.g., up to 24.3 IQ points gained if one embryo is selected out of 1000). If this process is iterated over many generations, the gains could be an order of magnitude improvement. Bostrom suggests that deriving new gametes from embryonic stem cells could be used to iterate the selection process rapidly.<sup>[18]</sup> A well-organized society of high-intelligence humans of this sort could potentially achieve collective superintelligence.<sup>[19]</sup> |
| # Novel architectures – Others suggest that new AI architectures, potentially inspired by neuroscience, may be necessary to achieve AGI and ASI.<ref>{{Cite journal |last1=Hassabis |first1=Demis |last2=Kumaran |first2=Dharshan |last3=Summerfield |first3=Christopher |last4=Botvinick |first4=Matthew |title=Neuroscience-Inspired Artificial Intelligence |journal=Neuron |volume=95 |issue=2 |year=2017 |pages=245–258 |doi=10.1016/j.neuron.2017.06.011|pmid=28728020 }}</ref>
| |
| # Hybrid systems – Combining different AI approaches, including symbolic AI and neural networks, could potentially lead to more robust and capable systems.<ref>{{Cite arXiv |last1=Garcez |first1=Artur d'Avila |last2=Lamb |first2=Luis C. |title=Neurosymbolic AI: The 3rd Wave |year=2020|class=cs.AI |eprint=2012.05876 }}</ref>
| |
|
| |
|
| === Computational advantages ===
| | Alternatively, collective intelligence might be constructed by better organizing humans at present levels of individual intelligence. Several writers have suggested that human civilization, or some aspect of it (e.g., the Internet, or the economy), is coming to function like a global brain with capacities far exceeding its component agents.<sup>[20]</sup> A prediction market is sometimes considered as an example of a working collective intelligence system, consisting of humans only (assuming algorithms are not used to inform decisions).<sup>[21]</sup> |
|
| |
|
| Artificial systems have several potential advantages over biological intelligence:
| | A final method of intelligence amplification would be to directly enhance individual humans, as opposed to enhancing their social or reproductive dynamics. This could be achieved using nootropics, somatic gene therapy, or brain−computer interfaces. However, Bostrom expresses skepticism about the scalability of the first two approaches and argues that designing a superintelligent cyborg interface is an AI-complete problem.<sup>[22]</sup> |
| | |
| # Speed – Computer components operate much faster than biological neurons. Modern microprocessors (~2 GHz) are seven orders of magnitude faster than neurons (~200 Hz).{{sfn|Bostrom|2014|p=59}}
| |
| # Scalability – AI systems can potentially be scaled up in size and computational capacity more easily than biological brains.
| |
| # Modularity – Different components of AI systems can be improved or replaced independently.
| |
| # Memory – AI systems can have perfect recall and vast knowledge bases. It is also much less constrained than humans when it comes to working memory.{{sfn|Bostrom|2014|p=59}}
| |
| # Multitasking – AI can perform multiple tasks simultaneously in ways not possible for biological entities.
| |
| | |
| === Potential path through transformer models ===
| |
| | |
| Recent advancements in transformer-based models have led some researchers to speculate that the path to ASI might lie in scaling up and improving these architectures. This view suggests that continued improvements in transformer models or similar architectures could lead directly to ASI.<ref>{{Cite journal |last1=Sutskever |first1=Ilya |title=A Brief History of Scaling |journal=ACM Queue |volume=21 |issue=4 |year=2023 |pages=31–43 |doi=10.1145/3595878.3605016|doi-broken-date=1 July 2025 }}</ref>
| |
| | |
| Some experts even argue that current large language models like GPT-4 may already exhibit early signs of AGI or ASI capabilities.<ref>{{Cite arXiv |last1=Bubeck |first1=Sébastien |last2=Chandrasekaran |first2=Varun |last3=Eldan |first3=Ronen |last4=Gehrke |first4=Johannes |last5=Horvitz |first5=Eric |last6=Kamar |first6=Ece |last7=Lee |first7=Peter |last8=Lee |first8=Yin Tat |last9=Li |first9=Yuanzhi |last10=Lundberg |first10=Scott |last11=Nori |first11=Harsha |last12=Palangi |first12=Hamid |last13=Precup |first13=Doina |last14=Sountsov |first14=Pavel |last15=Srivastava |first15=Sanjana |last16=Tessler |first16=Catherine |last17=Tian |first17=Jianfeng |last18=Zaheer |first18=Manzil |title=Sparks of Artificial General Intelligence: Early experiments with GPT-4 |date=22 March 2023|class=cs.CL |eprint=2303.12712 }}</ref> This perspective suggests that the transition from current AI to ASI might be more continuous and rapid than previously thought, blurring the lines between narrow AI, AGI, and ASI.
| |
| | |
| However, this view remains controversial. Critics argue that current models, while impressive, still lack crucial aspects of general intelligence such as true understanding, reasoning, and adaptability across diverse domains.<ref>{{Cite arXiv |last1=Marcus |first1=Gary |title=The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence |year=2020|class=cs.AI |eprint=2002.06177 }}</ref>
| |
| | |
| The debate over whether the path to ASI will involve a distinct AGI phase or a more direct scaling of current technologies remains ongoing, with significant implications for AI development strategies and safety considerations.
| |
| | |
| === Challenges and uncertainties ===
| |
| | |
| Despite these potential advantages, there are significant challenges and uncertainties in achieving ASI:
| |
| | |
| # [[Ethics of artificial intelligence|Ethical]] and [[AI safety|safety]] concerns – The development of ASI raises numerous ethical questions and potential risks that need to be addressed.{{sfn|Russell|2019}}
| |
| # Computational requirements – The computational resources required for ASI might be far beyond current capabilities.
| |
| # Fundamental limitations – There may be fundamental limitations to intelligence that apply to both artificial and biological systems.
| |
| # Unpredictability – The path to ASI and its consequences are highly uncertain and difficult to predict.
| |
| | |
| As research in AI continues to advance rapidly, the question of the feasibility of ASI remains a topic of intense debate and study in the scientific community.
| |
| | |
| ==Feasibility of biological superintelligence==
| |
| [[Carl Sagan]] suggested that the advent of [[Caesarean section]]s and [[in vitro fertilisation|''in vitro'' fertilization]] may permit humans to evolve larger heads, resulting in improvements via [[natural selection]] in the [[adaptive evolution in the human genome|heritable]] component of [[human intelligence]].<ref>{{cite book |last=Sagan |first=Carl |author-link=Carl Sagan |date=1977 |title=The Dragons of Eden |publisher=Random House |title-link=The Dragons of Eden }}</ref> By contrast, [[Gerald Crabtree]] has argued that decreased selection pressure is resulting in a slow, centuries-long [[Fertility and intelligence|reduction in human intelligence]] and that this process instead is likely to continue. There is no scientific consensus concerning either possibility and in both cases, the biological change would be slow, especially relative to rates of cultural change.
| |
| | |
| [[Selective breeding]], [[nootropics]], [[Epigenetics|epigenetic modulation]], and [[genetic engineering]] could improve human intelligence more rapidly. Bostrom writes that if we come to understand the genetic component of intelligence, pre-implantation genetic diagnosis could be used to select for embryos with as much as 4 points of IQ gain (if one embryo is selected out of two), or with larger gains (e.g., up to 24.3 IQ points gained if one embryo is selected out of 1000). If this process is iterated over many generations, the gains could be an order of magnitude improvement. Bostrom suggests that deriving new gametes from embryonic stem cells could be used to iterate the selection process rapidly.{{sfn|Bostrom|2014|pp=37–39}} A well-organized society of high-intelligence humans of this sort could potentially achieve [[collective intelligence|collective]] superintelligence.{{sfn|Bostrom|2014|p=39}}
| |
| | |
| Alternatively, collective intelligence might be constructional by better organizing humans at present levels of individual intelligence. Several writers have suggested that human civilization, or some aspect of it (e.g., the Internet, or the economy), is coming to function like a [[global brain]] with capacities far exceeding its component agents. If this systemic superintelligence relies heavily on artificial components, however, it may qualify as an AI rather than as a biology-based [[superorganism]].{{sfn|Bostrom|2014|pp=48–49}} A [[prediction market]] is sometimes considered as an example of a working collective intelligence system, consisting of humans only (assuming algorithms are not used to inform decisions).<ref>{{citation|title=Prediction Markets as an Aggregation Mechanism for Collective Intelligence|url=https://escholarship.org/uc/item/8mg0p0zc|year=2007|last=Watkins|first=Jennifer H.}}</ref>
| |
| | |
| A final method of intelligence amplification would be to directly [[Neuroenhancement|enhance]] individual humans, as opposed to enhancing their social or reproductive dynamics. This could be achieved using [[nootropics]], somatic [[gene therapy]], or [[brain–computer interface|brain−computer interface]]s. However, Bostrom expresses skepticism about the scalability of the first two approaches and argues that designing a superintelligent [[cyborg]] interface is an [[AI-complete]] problem.{{sfn|Bostrom|2014|pp=36–37, 42, 47}} | |
|
| |
|
| == Forecasts == | | == Forecasts == |
| Most surveyed AI researchers expect machines to eventually be able to rival humans in intelligence, though there is little consensus on when this will likely happen. At the 2006 [[AI@50]] conference, 18% of attendees reported expecting machines to be able "to simulate learning and every other aspect of human intelligence" by 2056; 41% of attendees expected this to happen sometime after 2056; and 41% expected machines to never reach that milestone.<ref>{{cite web|last1=Maker|first1=Meg Houston|title=AI@50: First Poll|url=http://www.megmaker.com/2006/07/ai50_first_poll.html|archive-url=https://web.archive.org/web/20140513052243/http://www.megmaker.com/2006/07/ai50_first_poll.html|archive-date=2014-05-13|date=July 13, 2006}}</ref> | | Most surveyed AI researchers expect machines to eventually be able to rival humans in intelligence, though there is little consensus on when this will likely happen.<sup>[23]</sup> |
|
| |
|
| In a survey of the 100 most cited authors in AI (as of May 2013, according to Microsoft academic search), the median year by which respondents expected machines "that can carry out most human professions at least as well as a typical human" (assuming no [[global catastrophic risks|global catastrophe]] occurs) with 10% confidence is 2024 (mean 2034, st. dev. 33 years), with 50% confidence is 2050 (mean 2072, st. dev. 110 years), and with 90% confidence is 2070 (mean 2168, st. dev. 342 years). These estimates exclude the 1.2% of respondents who said no year would ever reach 10% confidence, the 4.1% who said 'never' for 50% confidence, and the 16.5% who said 'never' for 90% confidence. Respondents assigned a median 50% probability to the possibility that machine superintelligence will be invented within 30 years of the invention of approximately human-level machine intelligence.{{sfn|Müller|Bostrom|2016|pp=3–4, 6, 9–12}} | | In a 2022 survey, the median year by which respondents expected "High-level machine intelligence" with 50% confidence is 2061. The survey defined the achievement of high-level machine intelligence as when unaided machines can accomplish every task better and more cheaply than human workers.<sup>[24]</sup> |
|
| |
|
| In a 2022 survey, the median year by which respondents expected "High-level machine intelligence" with 50% confidence is 2061. The survey defined the achievement of high-level machine intelligence as when unaided machines can accomplish every task better and more cheaply than human workers.<ref>{{Cite web |title=AI timelines: What do experts in artificial intelligence expect for the future? |url=https://ourworldindata.org/ai-timelines |access-date=2023-08-09 |website=Our World in Data|date=7 February 2023 |last1=Roser |first1=Max }}</ref> | | In 2023, OpenAI leaders Sam Altman, Greg Brockman and Ilya Sutskever published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years.<sup>[25]</sup> |
|
| |
|
| In 2023, [[OpenAI]] leaders [[Sam Altman]], [[Greg Brockman]] and [[Ilya Sutskever]] published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years.<ref>{{Cite web |title=Governance of superintelligence |url=https://openai.com/blog/governance-of-superintelligence |access-date=2023-05-30 |website=openai.com |language=en-US}}</ref> In 2024, Ilya Sutskever left OpenAI to cofound the startup [[Safe Superintelligence Inc.|Safe Superintelligence]], which focuses solely on creating a superintelligence that is [[AI safety|safe]] by design, while avoiding "distraction by management overhead or product cycles".<ref>{{Cite web |last=Vance |first=Ashlee |date=June 19, 2024 |title=Ilya Sutskever Has a New Plan for Safe Superintelligence |url=https://www.bloomberg.com/news/articles/2024-06-19/openai-co-founder-plans-new-ai-focused-research-lab |access-date=2024-06-19 |website=Bloomberg}}</ref> Despite still offering no product, the startup became valued at $30 billion in February 2025.<ref>{{Cite web |date=2025-02-24 |title=There's Something Very Weird About This $30 Billion AI Startup by a Man Who Said Neural Networks May Already Be Conscious |url=https://futurism.com/ilya-sutskever-safe-superintelligence-product |access-date=2025-04-27 |website=Futurism}}</ref> In 2025, the forecast scenario "AI 2027" led by [[Daniel Kokotajlo (researcher)|Daniel Kokotajlo]] predicted rapid progress in the automation of coding and AI research, followed by ASI.<ref>{{Cite news |last=Roose |first=Kevin |date=2025-04-03 |title=This A.I. Forecast Predicts Storms Ahead |url=https://www.nytimes.com/2025/04/03/technology/ai-futures-project-ai-2027.html |access-date=2025-04-27 |work=The New York Times |language=en-US |issn=0362-4331}}</ref>
| | In 2025, the forecast scenario AI 2027 led by Daniel Kokotajlo predicted rapid progress in the automation of coding and AI research, followed by ASI.<sup>[26]</sup> In September 2025, a review of surveys of scientists and industry experts from the last 15 years reported that most agreed that artificial general intelligence (AGI), a level well below technological singularity, will occur before the year 2100.<sup>[27]</sup> A more recent analysis by AIMultiple reported that, “Current surveys of AI researchers are predicting AGI around 2040”.<sup>[27]</sup> |
|
| |
|
| == Design considerations == | | == Design considerations == |
| | Exploring the potential motivations of an artificial superintelligence, Bostrom distinguishes final goals and instrumental goals. From the point of view of an agent, final goals are intrinsically valuable, whereas instrumental goals are only useful for attaining final goals. He proposed the "orthogonality thesis", which postulates that in principle, virtually any final goal can be combined with virtually any level of intelligence. Bostrom also introduced the concept of instrumental convergence, which postulates that certain instrumental goals (such as self-preservation, resource acquisition or cognitive enhancement) increase the probability of achieving final goals in a wide range of situations, and would thus likely be pursued by a broad spectrum of intelligent agents.<sup>[28]</sup> |
|
| |
|
| The design of superintelligent AI systems raises critical questions about what values and goals these systems should have. Several proposals have been put forward:{{sfn|Bostrom|2014|pp=209–221}}
| | William MacAskill argued that aligning superintelligence with current human values could be catastrophic if those values are permanently locked in and humanity still has moral blind spots like slavery in the past.<sup>[29]</sup> |
|
| |
|
| === Value alignment proposals ===
| | Several proposals for an ASI's final goals have been put forward:<sup>[30]</sup> |
|
| |
|
| * [[Coherent extrapolated volition]] (CEV) – The AI should have the values upon which humans would converge if they were more knowledgeable and rational.
| | Coherent extrapolated volition (CEV) – The AI should have the values upon which humans would converge if they were more knowledgeable and rational. |
| * Moral rightness (MR) – The AI should be programmed to do what is morally right, relying on its superior cognitive abilities to determine ethical actions.
| |
| * Moral permissibility (MP) – The AI should stay within the bounds of moral permissibility while otherwise pursuing goals aligned with human values (similar to CEV).
| |
|
| |
|
| Bostrom elaborates on these concepts:
| | Moral rightness (MR) – The AI should be programmed to do what is morally right, relying on its superior cognitive abilities to determine ethical actions. |
|
| |
|
| <blockquote>instead of implementing humanity's coherent extrapolated volition, one could try to build an AI to do what is morally right, relying on the AI's superior cognitive capacities to figure out just which actions fit that description. We can call this proposal "moral rightness" (MR){{nbsp}}...
| | Moral permissibility (MP) – The AI should stay within the bounds of moral permissibility while otherwise pursuing goals aligned with human values (similar to CEV). |
|
| |
|
| MR would also appear to have some disadvantages. It relies on the notion of "morally right", a notoriously difficult concept, one with which philosophers have grappled since antiquity without yet attaining consensus as to its analysis. Picking an erroneous explication of "moral rightness" could result in outcomes that would be morally very wrong{{nbsp}}...
| | Bostrom elaborates on these concepts: |
| | |
| One might try to preserve the basic idea of the MR model while reducing its demandingness by focusing on ''moral permissibility'': the idea being that we could let the AI pursue humanity's CEV so long as it did not act in morally impermissible ways.{{sfn|Bostrom|2014|pp=209–221}}</blockquote>
| |
| | |
| === Recent developments ===
| |
| | |
| Since Bostrom's analysis, new approaches to AI value alignment have emerged:
| |
| | |
| * Inverse Reinforcement Learning (IRL) – This technique aims to infer human preferences from observed behavior, potentially offering a more robust approach to value alignment.<ref>{{Cite journal |last1=Christiano |first1=Paul |last2=Leike |first2=Jan |last3=Brown |first3=Tom B. |last4=Martic |first4=Miljan |last5=Legg |first5=Shane |last6=Amodei |first6=Dario |date=2017 |title=Deep Reinforcement Learning from Human Preferences |url=https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf |journal=NeurIPS |arxiv=1706.03741}}</ref>
| |
| * [[Constitutional AI]] – Proposed by Anthropic, this involves training AI systems with explicit ethical principles and constraints.<ref>{{Cite web |date=December 15, 2022 |title=Constitutional AI: Harmlessness from AI Feedback |url=https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback |access-date= |website=Anthropic |language=en}}</ref>
| |
| * Debate and amplification – These techniques, explored by OpenAI, use AI-assisted debate and iterative processes to better understand and align with human values.<ref>{{Cite web |date=October 22, 2018 |title=Learning complex goals with iterated amplification |url=https://openai.com/index/learning-complex-goals-with-iterated-amplification/ |website=OpenAI}}</ref>
| |
| | |
| === Transformer LLMs and ASI ===
| |
| | |
| The rapid advancement of transformer-based LLMs has led to speculation about their potential path to ASI. Some researchers argue that scaled-up versions of these models could exhibit ASI-like capabilities:<ref>{{Cite web |date=2021 |title=On the Opportunities and Risks of Foundation Models |url=https://crfm.stanford.edu/report.html |website=Stanford University |arxiv=2108.07258 |last1=Bommasani |first1=Rishi |last2=Hudson |first2=Drew A. |last3=Adeli |first3=Ehsan |last4=Altman |first4=Russ |last5=Arora |first5=Simran |author6=Sydney von Arx |last7=Bernstein |first7=Michael S. |last8=Bohg |first8=Jeannette |last9=Bosselut |first9=Antoine |last10=Brunskill |first10=Emma |last11=Brynjolfsson |first11=Erik |last12=Buch |first12=Shyamal |last13=Card |first13=Dallas |last14=Castellon |first14=Rodrigo |last15=Chatterji |first15=Niladri |last16=Chen |first16=Annie |last17=Creel |first17=Kathleen |author18=Jared Quincy Davis |last19=Demszky |first19=Dora |last20=Donahue |first20=Chris |last21=Doumbouya |first21=Moussa |last22=Durmus |first22=Esin |last23=Ermon |first23=Stefano |last24=Etchemendy |first24=John |last25=Ethayarajh |first25=Kawin |last26=Fei-Fei |first26=Li |last27=Finn |first27=Chelsea |last28=Gale |first28=Trevor |last29=Gillespie |first29=Lauren |last30=Goel |first30=Karan |display-authors=1 }}</ref>
| |
| | |
| * Emergent abilities – As LLMs increase in size and complexity, they demonstrate unexpected capabilities not present in smaller models.<ref name=":0">{{Cite journal |date=2022-06-26 |title=Emergent Abilities of Large Language Models |journal=Transactions on Machine Learning Research |language=en |arxiv=2206.07682 |issn=2835-8856 |last1=Wei |first1=Jason |last2=Tay |first2=Yi |last3=Bommasani |first3=Rishi |last4=Raffel |first4=Colin |last5=Zoph |first5=Barret |last6=Borgeaud |first6=Sebastian |last7=Yogatama |first7=Dani |last8=Bosma |first8=Maarten |last9=Zhou |first9=Denny |last10=Metzler |first10=Donald |last11=Chi |first11=Ed H. |last12=Hashimoto |first12=Tatsunori |last13=Vinyals |first13=Oriol |last14=Liang |first14=Percy |last15=Dean |first15=Jeff |last16=Fedus |first16=William }}</ref>
| |
| * In-context learning – LLMs show the ability to adapt to new tasks without fine-tuning, potentially mimicking general intelligence.<ref>{{Cite journal |date=2020 |title=Language Models are Few-Shot Learners |journal=NeurIPS |arxiv=2005.14165 |last1=Brown |first1=Tom B. |last2=Mann |first2=Benjamin |last3=Ryder |first3=Nick |last4=Subbiah |first4=Melanie |last5=Kaplan |first5=Jared |last6=Dhariwal |first6=Prafulla |last7=Neelakantan |first7=Arvind |last8=Shyam |first8=Pranav |last9=Sastry |first9=Girish |last10=Askell |first10=Amanda |last11=Agarwal |first11=Sandhini |last12=Herbert-Voss |first12=Ariel |last13=Krueger |first13=Gretchen |last14=Henighan |first14=Tom |last15=Child |first15=Rewon |last16=Ramesh |first16=Aditya |last17=Ziegler |first17=Daniel M. |last18=Wu |first18=Jeffrey |last19=Winter |first19=Clemens |last20=Hesse |first20=Christopher |last21=Chen |first21=Mark |last22=Sigler |first22=Eric |last23=Litwin |first23=Mateusz |last24=Gray |first24=Scott |last25=Chess |first25=Benjamin |last26=Clark |first26=Jack |last27=Berner |first27=Christopher |last28=McCandlish |first28=Sam |last29=Radford |first29=Alec |last30=Sutskever |first30=Ilya |display-authors=1 }}</ref>
| |
| * Multi-modal integration – Recent models can process and generate various types of data, including text, images, and audio.<ref>{{Cite journal |date=2022 |title=Flamingo: a Visual Language Model for Few-Shot Learning |journal=NeurIPS|arxiv=2204.14198 |last1=Alayrac |first1=Jean-Baptiste |last2=Donahue |first2=Jeff |last3=Luc |first3=Pauline |last4=Miech |first4=Antoine |last5=Barr |first5=Iain |last6=Hasson |first6=Yana |last7=Lenc |first7=Karel |last8=Mensch |first8=Arthur |last9=Millican |first9=Katie |last10=Reynolds |first10=Malcolm |last11=Ring |first11=Roman |last12=Rutherford |first12=Eliza |last13=Cabi |first13=Serkan |last14=Han |first14=Tengda |last15=Gong |first15=Zhitao |last16=Samangooei |first16=Sina |last17=Monteiro |first17=Marianne |last18=Menick |first18=Jacob |last19=Borgeaud |first19=Sebastian |last20=Brock |first20=Andrew |last21=Nematzadeh |first21=Aida |last22=Sharifzadeh |first22=Sahand |last23=Binkowski |first23=Mikolaj |last24=Barreira |first24=Ricardo |last25=Vinyals |first25=Oriol |last26=Zisserman |first26=Andrew |last27=Simonyan |first27=Karen }}</ref>
| |
| | |
| However, critics argue that current LLMs lack true understanding and are merely sophisticated pattern matchers, raising questions about their suitability as a path to ASI.<ref>{{Cite journal |last=Marcus |first=Gary |date=August 11, 2022 |title=Deep Learning Alone Isn't Getting Us To Human-Like AI |url=https://www.noemamag.com/deep-learning-alone-isnt-getting-us-to-human-like-ai/ |website=Noema}}</ref>
| |
| | |
| === Other perspectives on artificial superintelligence ===
| |
| | |
| Additional viewpoints on the development and implications of superintelligence include:
| |
| | |
| * [[Recursive self-improvement]] – [[I. J. Good]] proposed the concept of an "intelligence explosion", where an AI system could rapidly improve its own intelligence, potentially leading to superintelligence.<ref>{{Cite news |date=March 30, 2017 |title=The AI apocalypse: will the human race soon be terminated? |url=https://www.irishtimes.com/business/innovation/the-ai-apocalypse-will-the-human-race-soon-be-terminated-1.3019220 |access-date= |newspaper=The Irish Times |language=en}}</ref>
| |
| * Orthogonality thesis – Bostrom argues that an AI's level of intelligence is orthogonal to its final goals, meaning a superintelligent AI could have any set of motivations.<ref>{{Cite journal |last=Bostrom |first=Nick |date=2012 |title=The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents |url=https://nickbostrom.com/superintelligentwill.pdf |journal=Minds and Machines|volume=22 |issue=2 |pages=71–85 |doi=10.1007/s11023-012-9281-3 }}</ref>
| |
| * [[Instrumental convergence]] – Certain instrumental goals (e.g., self-preservation, resource acquisition) might be pursued by a wide range of AI systems, regardless of their final goals.<ref>{{Cite journal |last=Omohundro |first=Stephen M. |date=January 2008 |title=The basic AI drives |url=https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf |journal=Frontiers in Artificial Intelligence and Applications}}</ref>
| |
| | |
| === Challenges and ongoing research ===
| |
|
| |
|
| The pursuit of value-aligned AI faces several challenges:
| | instead of implementing humanity's coherent extrapolated volition, one could try to build an AI to do what is morally right, relying on the AI's superior cognitive capacities to figure out just which actions fit that description. We can call this proposal "moral rightness" (MR) ... |
|
| |
|
| * Philosophical uncertainty in defining concepts like "moral rightness"
| | MR would also appear to have some disadvantages. It relies on the notion of "morally right", a notoriously difficult concept, one with which philosophers have grappled since antiquity without yet attaining consensus as to its analysis. Picking an erroneous explication of "moral rightness" could result in outcomes that would be morally very wrong ... |
| * Technical complexity in translating ethical principles into precise algorithms
| |
| * Potential for unintended consequences even with well-intentioned approaches
| |
|
| |
|
| Current research directions include multi-stakeholder approaches to incorporate diverse perspectives, developing methods for scalable oversight of AI systems, and improving techniques for robust value learning.<ref>{{Cite journal |last=Gabriel |first=Iason |date=2020-09-01 |title=Artificial Intelligence, Values, and Alignment |journal=Minds and Machines |language=en |volume=30 |issue=3 |pages=411–437 |doi=10.1007/s11023-020-09539-2 |issn=1572-8641|doi-access=free |arxiv=2001.09768 }}</ref>{{sfn|Russell|2019}}
| | One might try to preserve the basic idea of the MR model while reducing its demandingness by focusing on ''moral permissibility'': the idea being that we could let the AI pursue humanity's CEV so long as it did not act in morally impermissible ways.<sup>[30]</sup> |
|
| |
|
| Al research is rapidly progressing towards superintelligence. Addressing these design challenges remains crucial for creating ASI systems that are both powerful and aligned with human interests.
| | == See also == |
| | AI takeover |
|
| |
|
| == Potential threat to humanity ==
| | Artificial brain |
| {{Main|Existential risk from artificial general intelligence|AI alignment|AI safety}}
| |
|
| |
|
| The development of artificial superintelligence (ASI) has raised concerns about potential existential risks to humanity. Researchers have proposed various scenarios in which an ASI could pose a significant threat:
| | Artificial intelligence arms race |
|
| |
|
| === Intelligence explosion and control problem ===
| | Effective altruism |
|
| |
|
| Some researchers argue that through recursive self-improvement, an ASI could rapidly become so powerful as to be beyond human control. This concept, known as an "intelligence explosion", was first proposed by I. J. Good in 1965:
| | Ethics of artificial intelligence |
|
| |
|
| {{quote|Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.<ref>{{Cite journal |last=Good |first=I. J. |date=1965 |title=Speculations Concerning the First Ultraintelligent Machine |journal=Advances in Computers}}</ref>}}
| | Existential risk |
|
| |
|
| This scenario presents the AI control problem: how to create an ASI that will benefit humanity while avoiding unintended harmful consequences.{{sfn|Russell|2019|pp=137-160}} Eliezer Yudkowsky argues that solving this problem is crucial before ASI is developed, as a superintelligent system might be able to thwart any subsequent attempts at control.<ref>{{Cite journal |last=Yudkowsky |first=Eliezer |date=2008 |title=Artificial Intelligence as a Positive and Negative Factor in Global Risk |url=https://intelligence.org/files/AIPosNegFactor.pdf |journal=Global Catastrophic Risks|doi=10.1093/oso/9780198570509.003.0021 |isbn=978-0-19-857050-9 }}</ref>
| | Friendly artificial intelligence |
|
| |
|
| === Unintended consequences and goal misalignment ===
| | Future of Humanity Institute |
|
| |
|
| Even with benign intentions, an ASI could potentially cause harm due to misaligned goals or unexpected interpretations of its objectives. Nick Bostrom provides a stark example of this risk:
| | Intelligent agent |
|
| |
|
| {{quote|When we create the first superintelligent entity, we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question.{{sfn|Bostrom|2002}}}}
| | Machine ethics |
|
| |
|
| Stuart Russell offers another illustrative scenario:
| | Machine Intelligence Research Institute |
|
| |
|
| {{quote|A system given the objective of maximizing human happiness might find it easier to rewire human neurology so that humans are always happy regardless of their circumstances, rather than to improve the external world.{{sfn|Russell|2019|p=136}}}}
| | Machine learning |
|
| |
|
| These examples highlight the potential for catastrophic outcomes even when an ASI is not explicitly designed to be harmful, underscoring the critical importance of precise goal specification and alignment.
| | Neural scaling law – Statistical law in machine learning |
|
| |
|
| === Potential mitigation strategies ===
| | Noosphere – Philosophical concept of biosphere successor via humankind's rational activities |
|
| |
|
| Researchers have proposed various approaches to mitigate risks associated with ASI:
| | Outline of artificial intelligence |
|
| |
|
| * [[AI capability control|Capability control]] – Limiting an ASI's ability to influence the world, such as through physical isolation or restricted access to resources.{{sfn|Bostrom|2014|pp=129-136}}
| | Posthumanism |
| * Motivational control – Designing ASIs with goals that are fundamentally aligned with human values.{{sfn|Bostrom|2014|pp=136-143}}
| |
| * Ethical AI – Incorporating ethical principles and decision-making frameworks into ASI systems.<ref>{{Cite book |last1=Wallach |first1=Wendell |url=https://books.google.com/books?id=_r3N82ETng4C |title=Moral Machines: Teaching Robots Right from Wrong |last2=Allen |first2=Colin |date=2008-11-19 |publisher=Oxford University Press |isbn=978-0-19-970596-2 |language=en}}</ref>
| |
| * Oversight and governance – Developing robust international frameworks for the development and deployment of ASI technologies.<ref name=":1">{{Cite web |last=Dafoe |first=Allan |date=August 27, 2018 |title=AI Governance: A Research Agenda |url=https://cdn.governance.ai/GovAI-Research-Agenda.pdf |website=Center for the Governance of AI}}</ref>
| |
|
| |
|
| Despite these proposed strategies, some experts, such as Roman Yampolskiy, argue that the challenge of controlling a superintelligent AI might be fundamentally unsolvable, emphasizing the need for extreme caution in ASI development.<ref>{{Cite web |last=Yampolskiy |first=Roman V. |date=July 18, 2020 |title=On Controllability of Artificial Intelligence |url=https://philpapers.org/archive/YAMOCO.pdf |arxiv=2008.04071}}</ref>
| | Robotics |
|
| |
|
| === Debate and skepticism ===
| | Self-replication |
|
| |
|
| Not all researchers agree on the likelihood or severity of ASI-related existential risks. Some, like [[Rodney Brooks]], argue that fears of superintelligent AI are overblown and based on unrealistic assumptions about the nature of intelligence and technological progress.<ref>{{Cite web |last=Brooks |first=Rodney |date=October 6, 2017 |title=The Seven Deadly Sins of AI Predictions |url=https://www.technologyreview.com/2017/10/06/241837/the-seven-deadly-sins-of-ai-predictions/ |access-date=2024-10-23 |website=MIT Technology Review |language=en}}</ref> Others, such as [[Joanna Bryson]], contend that [[Anthropomorphism|anthropomorphizing]] AI systems leads to misplaced concerns about their potential threats.<ref>{{Cite journal |last=Bryson |first=Joanna J |date=2019 |title=The Past Decade and Future of AI's Impact on Society |url=https://researchportal.bath.ac.uk/en/publications/the-past-decade-and-future-of-ais-impact-on-society |journal=Towards a New Enlightenment? A Transcendent Decade|volume=11 |isbn=978-84-17141-21-9 }}</ref>
| | Self-replicating machine |
|
| |
|
| === Recent developments and current perspectives ===
| | Superintelligence: Paths, Dangers, Strategies |
| | |
| The rapid advancement of LLMs and other AI technologies has intensified debates about the proximity and potential risks of ASI. While there is no scientific consensus, some researchers and AI practitioners argue that current AI systems may already be approaching AGI or even ASI capabilities.
| |
| | |
| * LLM capabilities – Recent LLMs like GPT-4 have demonstrated unexpected abilities in areas such as reasoning, problem-solving, and multi-modal understanding, leading some to speculate about their potential path to ASI.<ref>{{Cite arXiv |date=April 2023 |title=Sparks of Artificial General Intelligence: Early experiments with GPT-4 |eprint=2303.12712 |last1=Bubeck |first1=Sébastien |last2=Chandrasekaran |first2=Varun |last3=Eldan |first3=Ronen |last4=Gehrke |first4=Johannes |last5=Horvitz |first5=Eric |last6=Kamar |first6=Ece |last7=Lee |first7=Peter |author8=Yin Tat Lee |last9=Li |first9=Yuanzhi |last10=Lundberg |first10=Scott |last11=Nori |first11=Harsha |last12=Palangi |first12=Hamid |author13=Marco Tulio Ribeiro |last14=Zhang |first14=Yi |class=cs.CL }}</ref>
| |
| * Emergent behaviors – Studies have shown that as AI models increase in size and complexity, they can exhibit emergent capabilities not present in smaller models, potentially indicating a trend towards more general intelligence.<ref name=":0" />
| |
| * Rapid progress – The pace of AI advancement has led some to argue that we may be closer to ASI than previously thought, with potential implications for existential risk.<ref>{{Cite book |last=Ord |first=Toby |title=The precipice: existential risk and the future of humanity |date=2020 |publisher=Bloomsbury academic |isbn=978-1-5266-0023-3 |location=london New York (N.Y.)}}</ref>
| |
| | |
| As of 2024, AI skeptics such as [[Gary Marcus]] caution against premature claims of AGI or ASI, arguing that current AI systems, despite their impressive capabilities, still lack true understanding and general intelligence.<ref>{{Cite web |date=17 September 2024 |title=How and Why Gary Marcus Became AI's Leading Critic > Marcus says generative AI like ChatGPT poses immediate dangers |url=https://spectrum.ieee.org/gary-marcus |access-date= |website=IEEE Spectrum |language=en}}</ref> They emphasize the significant challenges that remain in achieving human-level intelligence, let alone superintelligence.
| |
| | |
| The debate surrounding the current state and trajectory of AI development underscores the importance of continued research into AI safety and ethics, as well as the need for robust governance frameworks to manage potential risks as AI capabilities continue to advance.<ref name=":1" />
| |
| | |
| ==See also==
| |
| {{div col}}
| |
| * [[Artificial general intelligence]]
| |
| * [[AI safety]]
| |
| * [[AI takeover]]
| |
| * [[Artificial brain]]
| |
| * [[Artificial intelligence arms race]]
| |
| * [[Effective altruism#Long-term future and global catastrophic risks|Effective altruism]]
| |
| * [[Ethics of artificial intelligence]]
| |
| * [[Existential risk]]
| |
| * [[Friendly artificial intelligence]]
| |
| * [[Future of Humanity Institute]]
| |
| * [[Intelligent agent]]
| |
| * [[Machine ethics]]
| |
| * [[Machine Intelligence Research Institute]]
| |
| * [[Machine learning]]
| |
| * {{Annotated link|Neural scaling law}}
| |
| *{{annotated link|Noosphere}}
| |
| * [[Outline of artificial intelligence]]
| |
| * [[Posthumanism]]
| |
| * [[Robotics]]
| |
| * [[Self-replication]]
| |
| * [[Self-replicating machine]]
| |
| * ''[[Superintelligence: Paths, Dangers, Strategies]]''
| |
| {{div col end}}
| |
|
| |
|
| ==References== | | ==References== |
| {{reflist}}
| |
|
| |
| === Papers ===
| |
| * {{Citation
| |
| | first = Nick
| |
| | last = Bostrom
| |
| | author-link=Nick Bostrom
| |
| | title=Existential Risks
| |
| | journal=[[Journal of Evolution and Technology]]
| |
| | year=2002
| |
| | volume=9
| |
| | url=http://www.nickbostrom.com/existential/risks.html
| |
| | access-date=2007-08-07
| |
| }}.
| |
| * {{Cite journal |last=Chalmers |first=David |author-link=David Chalmers |date=2010 |title=The Singularity: A Philosophical Analysis |url=http://consc.net/papers/singularity.pdf |journal=Journal of Consciousness Studies |volume=17 |pages=7–65 }}
| |
| * {{Cite thesis |last=Legg |first=Shane |title=Machine Super Intelligence |type=PhD |url=http://www.vetta.org/documents/Machine_Super_Intelligence.pdf |year=2008 |publisher=Department of Informatics, University of Lugano |access-date=September 19, 2014 }}
| |
| * {{cite encyclopedia|year=2016|title=Fundamental Issues of Artificial Intelligence|publisher=Springer|chapter-url=http://www.philpapers.org/archive/MLLFPI|last1=Müller|first1=Vincent C. |author-link=Vincent C. Müller|editor-last=Müller|editor-first=Vincent C.|pages=553–571|last2=Bostrom|first2=Nick|author-link2=Nick Bostrom|chapter=Future Progress in Artificial Intelligence: A Survey of Expert Opinion}}
| |
| * {{Cite journal |last=Santos-Lang |first=Christopher | date=2014 |title=Our responsibility to manage evaluative diversity |url=http://grinfree.com/Responsibility.pdf |archive-url=https://web.archive.org/web/20140729094949/http://grinfree.com/Responsibility.pdf |url-status=usurped |archive-date=July 29, 2014 |journal=ACM SIGCAS Computers and Society |volume=44 |issue=2 |pages=16–19 |doi=10.1145/2656870.2656874 |s2cid=5649158 }}
| |
|
| |
| === Books ===
| |
| * {{Cite book |last=Hibbard |first=Bill |author-link=Bill Hibbard |date=2002 |title=Super-Intelligent Machines |publisher=Kluwer Academic/Plenum Publishers}}
| |
| * {{Cite book |last=Bostrom |first=Nick |author-link=Nick Bostrom |date=2014 |title=Superintelligence: Paths, Dangers, Strategies |publisher=Oxford University Press |title-link=Superintelligence: Paths, Dangers, Strategies}}
| |
| * {{Cite book |last=Tegmark |first=Max |author-link=Max Tegmark |title=[[Life 3.0|Life 3.0: being human in the age of artificial intelligence]] |date=2018 |isbn=978-0-14-198180-2 |location=London, England |language=en |oclc=1018461467}}
| |
| * {{Cite book |last=Russell |first=Stuart J. |title=[[Human Compatible|Human compatible: artificial intelligence and the problem of control]] |date=2019 |isbn=978-0-525-55861-3 |location=New York |oclc=1113410915}}
| |
| * {{Cite book |last=Sanders |first=Nada R. |author-link=Nada Sanders |title=The humachine: humankind, machines, and the future of enterprise |date=2020 |others=John D. Wood |isbn=978-0-429-00117-8 |edition=First |location=New York, New York |language=en |oclc=1119391268}}
| |
|
| |
| == External links ==
| |
| * ''[https://evolutionnews.org/2015/02/bill_gates_join/ Bill Gates Joins Stephen Hawking in Fears of a Coming Threat from "Superintelligence"]''
| |
| * ''[https://reason.com/2014/09/12/will-superintelligent-machines-destroy-h/ Will Superintelligent Machines Destroy Humanity?]''
| |
| * ''[https://www.ign.com/articles/2015/03/25/apple-co-founder-has-sense-of-foreboding-about-artificial-superintelligence Apple Co-founder Has Sense of Foreboding About Artificial Superintelligence]''
| |
|
| |
|
| {{Existential risk from artificial intelligence}}
| | # Mucci, Tim; Stryker, Cole (2023-12-14). "What Is Artificial Superintelligence?". ''IBM''. Retrieved 2025-04-17. |
| | # Bostrom 2014, Chapter 2. |
| | # Pearce, David (2012), "The Biointelligence Explosion: How Recursively Self-Improving Organic Robots will Modify their Own Source Code and Bootstrap Our Way to Full-Spectrum Superintelligence", in Eden, Amnon H.; Moor, James H.; Søraker, Johnny H.; Steinhart, Eric (eds.), ''Singularity Hypotheses'', The Frontiers Collection, Berlin, Germany and Heidelberg, Germany: Springer Berlin Heidelberg, pp. 199–238, doi:10.1007/978-3-642-32560-1_11, ISBN 978-3-642-32559-5, retrieved 2022-01-16 |
| | # Gouveia, Steven S., ed. (2020). "ch. 4, "Humans and Intelligent Machines: Co-evolution, Fusion or Replacement?", David Pearce". ''The Age of Artificial Intelligence: An Exploration''. Vernon Press. ISBN 978-1-62273-872-4. |
| | # Legg 2008, pp. 135–137. |
| | # Chalmers 2010, p. 7. |
| | # Chalmers 2010, pp. 7–9. |
| | # Chalmers 2010, pp. 10–11. |
| | # Chalmers 2010, pp. 11–13. |
| | # "Clever cogs". ''The Economist''. ISSN 0013-0613. Retrieved 2023-08-10. |
| | # Bostrom 2014, p. 59. |
| | # Bostrom 2014, pp. 56–57. |
| | # Bostrom 2014, pp. 52, 59–61. |
| | # Vance, Ashlee (June 19, 2024). "Ilya Sutskever Has a New Plan for Safe Superintelligence". ''Bloomberg''. Retrieved 2024-06-19. |
| | # "There's Something Very Weird About This $30 Billion AI Startup by a Man Who Said Neural Networks May Already Be Conscious". ''Futurism''. 2025-02-24. Retrieved 2025-04-27. |
| | # Tan, Eli (2025-12-10). "Meta's New A.I. Superstars Are Chafing Against the Rest of the Company". ''The New York Times''. ISSN 0362-4331. Retrieved 2026-02-02. |
| | # Sagan, Carl (1977). ''The Dragons of Eden''. Random House. |
| | # Bostrom 2014, pp. 37–39. |
| | # Bostrom 2014, p. 39. |
| | # Bostrom 2014, pp. 48–49. |
| | # Watkins, Jennifer H. (2007), Prediction Markets as an Aggregation Mechanism for Collective Intelligence |
| | # Bostrom 2014, pp. 36–37, 42, 47. |
| | # Roser, Max (2023-02-07). "AI timelines: What do experts in artificial intelligence expect for the future?". ''Our World in Data''. |
| | # Roser, Max (7 February 2023). "AI timelines: What do experts in artificial intelligence expect for the future?". ''Our World in Data''. Retrieved 2023-08-09. |
| | # "Governance of superintelligence". ''openai.com''. Retrieved 2023-05-30. |
| | # Roose, Kevin (2025-04-03). "This A.I. Forecast Predicts Storms Ahead". ''The New York Times''. ISSN 0362-4331. Retrieved 2025-04-27. |
| | # Orf, Darren (October 2025). "Humanity May Achieve the Singularity Within the Next 3 Months, Scientists Suggest". ''www.msn.com''. Popular Mechanics. Retrieved 2025-10-02. |
| | # Bostrom 2014, Chapter 7: The superintelligent will. |
| | # Piper, Kelsey (November 2022). "Review | What We Owe The Future". ''Asterisk Magazine''. Retrieved 2026-02-16. |
| | # Bostrom 2014, pp. 209–221. |
|
| |
|
| [[Category:Hypothetical technology]] | | [[Kategorie:Hypothetical technology]] |
| [[Category:Singularitarianism]] | | [[Kategorie:Singularitarianism]] |
| [[Category:Intelligence]] | | [[Kategorie:Intelligence]] |
| [[Category:Existential risk from artificial intelligence]] | | [[Kategorie:Existential risk from artificial intelligence]] |
A superintelligence is a hypothetical agent that possesses intelligence surpassing that of the most gifted human minds.[1] Philosopher Nick Bostrom defines superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest".[2]
Technological researchers disagree about how likely present-day human intelligence is to be surpassed. Some argue that advances in artificial intelligence (AI) will probably result in general reasoning systems that lack human cognitive limitations. Others believe that humans will evolve or directly modify their biology to achieve radically greater intelligence.[3][4] Several future study scenarios combine elements from both of these possibilities, suggesting that humans are likely to interface with computers, or upload their minds to computers, in a way that enables substantial intelligence amplification. The hypothetical creation of the first superintelligence may or may not result from an intelligence explosion or a technological singularity.
Some researchers believe that superintelligence will likely follow shortly after the development of artificial general intelligence. The first generally intelligent machines are likely to immediately hold an enormous advantage in at least some forms of mental capability, including the capacity of perfect recall, a vastly superior knowledge base, and the ability to multitask in ways not possible to biological entities.
Several scientists and forecasters have been arguing for prioritizing early research into the possible benefits and risks of human and machine cognitive enhancement, because of the potential social impact of such technologies.[5]
Artificial superintelligence
Philosopher David Chalmers argues that artificial general intelligence is a very likely path to artificial superintelligence (ASI). Chalmers breaks this claim down into an argument that AI can achieve equivalence to human intelligence, that it can be extended to surpass human intelligence, and that it can be further amplified to completely dominate humans across arbitrary tasks.[6]
Concerning human-level equivalence, Chalmers argues that the human brain is a mechanical system, and therefore ought to be emulatable by synthetic materials.[7] He also notes that human intelligence was able to biologically evolve, making it more likely that human engineers will be able to recapitulate this invention. Evolutionary algorithms, in particular, should be able to produce human-level AI.[8] Concerning intelligence extension and amplification, Chalmers argues that new AI technologies can generally be improved on, and that this is particularly likely when the invention can assist in designing new technologies.[9]
An AI system capable of self-improvement could enhance its own intelligence, thereby becoming more efficient at improving itself. This cycle of "recursive self-improvement" might cause an intelligence explosion, resulting in the creation of a superintelligence.[10]
Computer components already greatly surpass human performance in speed. Bostrom writes, "Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~2 GHz)."[11] Moreover, neurons transmit spike signals across axons at no greater than 120 m/s, "whereas existing electronic processing cores can communicate optically at the speed of light". Thus, the simplest example of a superintelligence may be an emulated human mind running on much faster hardware than the brain. A human-like reasoner who could think millions of times faster than current humans would have a dominant advantage in most reasoning tasks, particularly ones that require haste or long strings of actions.
Another advantage of computers is modularity, that is, their size or computational capacity can be increased. A non-human (or modified human) brain could become much larger than a present-day human brain, like many supercomputers. Bostrom also raises the possibility of collective superintelligence: a large enough number of separate reasoning systems, if they communicated and coordinated well enough, could act in aggregate with far greater capabilities than any sub-agent.
Humans outperform non-human animals in large part because of new or enhanced reasoning capacities, such as long-term planning and language use. (See evolution of human intelligence and primate cognition.) If there are other possible improvements to reasoning that would have a similarly large impact, this makes it more likely that an agent can be built that outperforms humans in the same fashion humans outperform chimpanzees.[12]
The above advantages hold for artificial superintelligence, but it is not clear how many hold for biological superintelligence. Physiological constraints limit the speed and size of biological brains in many ways that are inapplicable to machine intelligence. As such, writers on superintelligence have devoted much more attention to superintelligent AI scenarios.[13]
Projects
In 2024, Ilya Sutskever left OpenAI to cofound the startup Safe Superintelligence, which focuses solely on creating a superintelligence that is safe by design, while avoiding "distraction by management overhead or product cycles".[14] Despite still offering no product, the startup became valued at $30 billion in February 2025.[15]
In 2025, Meta created Meta Superintelligence Labs, a new AI division led by Alexandr Wang.[16]
Biological superintelligence
Carl Sagan suggested that the advent of Caesarean sections and in vitro fertilization may permit humans to evolve larger heads, resulting in improvements via natural selection in the heritable component of human intelligence.[17] By contrast, Gerald Crabtree has argued that decreased selection pressure is resulting in a slow, centuries-long reduction in human intelligence and that this process instead is likely to continue. There is no scientific consensus concerning either possibility and in both cases, the biological change would be slow, especially relative to rates of cultural change.
Selective breeding, nootropics, epigenetic modulation, and genetic engineering could improve human intelligence more rapidly. Bostrom writes that if we come to understand the genetic component of intelligence, pre-implantation genetic diagnosis could be used to select for embryos with as much as 4 points of IQ gain (if one embryo is selected out of two), or with larger gains (e.g., up to 24.3 IQ points gained if one embryo is selected out of 1000). If this process is iterated over many generations, the gains could be an order of magnitude improvement. Bostrom suggests that deriving new gametes from embryonic stem cells could be used to iterate the selection process rapidly.[18] A well-organized society of high-intelligence humans of this sort could potentially achieve collective superintelligence.[19]
Alternatively, collective intelligence might be constructed by better organizing humans at present levels of individual intelligence. Several writers have suggested that human civilization, or some aspect of it (e.g., the Internet, or the economy), is coming to function like a global brain with capacities far exceeding its component agents.[20] A prediction market is sometimes considered as an example of a working collective intelligence system, consisting of humans only (assuming algorithms are not used to inform decisions).[21]
A final method of intelligence amplification would be to directly enhance individual humans, as opposed to enhancing their social or reproductive dynamics. This could be achieved using nootropics, somatic gene therapy, or brain−computer interfaces. However, Bostrom expresses skepticism about the scalability of the first two approaches and argues that designing a superintelligent cyborg interface is an AI-complete problem.[22]
Forecasts
Most surveyed AI researchers expect machines to eventually be able to rival humans in intelligence, though there is little consensus on when this will likely happen.[23]
In a 2022 survey, the median year by which respondents expected "High-level machine intelligence" with 50% confidence is 2061. The survey defined the achievement of high-level machine intelligence as when unaided machines can accomplish every task better and more cheaply than human workers.[24]
In 2023, OpenAI leaders Sam Altman, Greg Brockman and Ilya Sutskever published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years.[25]
In 2025, the forecast scenario AI 2027 led by Daniel Kokotajlo predicted rapid progress in the automation of coding and AI research, followed by ASI.[26] In September 2025, a review of surveys of scientists and industry experts from the last 15 years reported that most agreed that artificial general intelligence (AGI), a level well below technological singularity, will occur before the year 2100.[27] A more recent analysis by AIMultiple reported that, “Current surveys of AI researchers are predicting AGI around 2040”.[27]
Design considerations
Exploring the potential motivations of an artificial superintelligence, Bostrom distinguishes final goals and instrumental goals. From the point of view of an agent, final goals are intrinsically valuable, whereas instrumental goals are only useful for attaining final goals. He proposed the "orthogonality thesis", which postulates that in principle, virtually any final goal can be combined with virtually any level of intelligence. Bostrom also introduced the concept of instrumental convergence, which postulates that certain instrumental goals (such as self-preservation, resource acquisition or cognitive enhancement) increase the probability of achieving final goals in a wide range of situations, and would thus likely be pursued by a broad spectrum of intelligent agents.[28]
William MacAskill argued that aligning superintelligence with current human values could be catastrophic if those values are permanently locked in and humanity still has moral blind spots like slavery in the past.[29]
Several proposals for an ASI's final goals have been put forward:[30]
Coherent extrapolated volition (CEV) – The AI should have the values upon which humans would converge if they were more knowledgeable and rational.
Moral rightness (MR) – The AI should be programmed to do what is morally right, relying on its superior cognitive abilities to determine ethical actions.
Moral permissibility (MP) – The AI should stay within the bounds of moral permissibility while otherwise pursuing goals aligned with human values (similar to CEV).
Bostrom elaborates on these concepts:
instead of implementing humanity's coherent extrapolated volition, one could try to build an AI to do what is morally right, relying on the AI's superior cognitive capacities to figure out just which actions fit that description. We can call this proposal "moral rightness" (MR) ...
MR would also appear to have some disadvantages. It relies on the notion of "morally right", a notoriously difficult concept, one with which philosophers have grappled since antiquity without yet attaining consensus as to its analysis. Picking an erroneous explication of "moral rightness" could result in outcomes that would be morally very wrong ...
One might try to preserve the basic idea of the MR model while reducing its demandingness by focusing on moral permissibility: the idea being that we could let the AI pursue humanity's CEV so long as it did not act in morally impermissible ways.[30]
See also
AI takeover
Artificial brain
Artificial intelligence arms race
Effective altruism
Ethics of artificial intelligence
Existential risk
Friendly artificial intelligence
Future of Humanity Institute
Intelligent agent
Machine ethics
Machine Intelligence Research Institute
Machine learning
Neural scaling law – Statistical law in machine learning
Noosphere – Philosophical concept of biosphere successor via humankind's rational activities
Outline of artificial intelligence
Posthumanism
Robotics
Self-replication
Self-replicating machine
Superintelligence: Paths, Dangers, Strategies
References
- Mucci, Tim; Stryker, Cole (2023-12-14). "What Is Artificial Superintelligence?". IBM. Retrieved 2025-04-17.
- Bostrom 2014, Chapter 2.
- Pearce, David (2012), "The Biointelligence Explosion: How Recursively Self-Improving Organic Robots will Modify their Own Source Code and Bootstrap Our Way to Full-Spectrum Superintelligence", in Eden, Amnon H.; Moor, James H.; Søraker, Johnny H.; Steinhart, Eric (eds.), Singularity Hypotheses, The Frontiers Collection, Berlin, Germany and Heidelberg, Germany: Springer Berlin Heidelberg, pp. 199–238, doi:10.1007/978-3-642-32560-1_11, ISBN 978-3-642-32559-5, retrieved 2022-01-16
- Gouveia, Steven S., ed. (2020). "ch. 4, "Humans and Intelligent Machines: Co-evolution, Fusion or Replacement?", David Pearce". The Age of Artificial Intelligence: An Exploration. Vernon Press. ISBN 978-1-62273-872-4.
- Legg 2008, pp. 135–137.
- Chalmers 2010, p. 7.
- Chalmers 2010, pp. 7–9.
- Chalmers 2010, pp. 10–11.
- Chalmers 2010, pp. 11–13.
- "Clever cogs". The Economist. ISSN 0013-0613. Retrieved 2023-08-10.
- Bostrom 2014, p. 59.
- Bostrom 2014, pp. 56–57.
- Bostrom 2014, pp. 52, 59–61.
- Vance, Ashlee (June 19, 2024). "Ilya Sutskever Has a New Plan for Safe Superintelligence". Bloomberg. Retrieved 2024-06-19.
- "There's Something Very Weird About This $30 Billion AI Startup by a Man Who Said Neural Networks May Already Be Conscious". Futurism. 2025-02-24. Retrieved 2025-04-27.
- Tan, Eli (2025-12-10). "Meta's New A.I. Superstars Are Chafing Against the Rest of the Company". The New York Times. ISSN 0362-4331. Retrieved 2026-02-02.
- Sagan, Carl (1977). The Dragons of Eden. Random House.
- Bostrom 2014, pp. 37–39.
- Bostrom 2014, p. 39.
- Bostrom 2014, pp. 48–49.
- Watkins, Jennifer H. (2007), Prediction Markets as an Aggregation Mechanism for Collective Intelligence
- Bostrom 2014, pp. 36–37, 42, 47.
- Roser, Max (2023-02-07). "AI timelines: What do experts in artificial intelligence expect for the future?". Our World in Data.
- Roser, Max (7 February 2023). "AI timelines: What do experts in artificial intelligence expect for the future?". Our World in Data. Retrieved 2023-08-09.
- "Governance of superintelligence". openai.com. Retrieved 2023-05-30.
- Roose, Kevin (2025-04-03). "This A.I. Forecast Predicts Storms Ahead". The New York Times. ISSN 0362-4331. Retrieved 2025-04-27.
- Orf, Darren (October 2025). "Humanity May Achieve the Singularity Within the Next 3 Months, Scientists Suggest". www.msn.com. Popular Mechanics. Retrieved 2025-10-02.
- Bostrom 2014, Chapter 7: The superintelligent will.
- Piper, Kelsey (November 2022). "Review | What We Owe The Future". Asterisk Magazine. Retrieved 2026-02-16.
- Bostrom 2014, pp. 209–221.
