argumate:

the-grey-tribe:

Imagine writing an AI (AI’) that is barely not smart enough to improve upon itself. Give it access to its own source code and tell it to rewrite itself. At first, it will make itself marginally stupider (AI-). But then, the mistakes that AI’ wiol have made in writing AI- will enable AI- to make even bigger mistakes, leading to a version AI–. These mistakes will add up. Within a few generations, AI– will become so stupid that no human being can understand it any more.

an intelligence implosion!