Will the future provide us with a genetically preprogrammed blissful paradise, or a global catastrophe? Will there be cessation of all suffering, or annihilation of all sentient life?
The history of futurology is not encouraging. Most “predictions” by futurists are more akin to prophecies that reveal more about personality, preoccupations and capacity for wish-fulfillment of the author than the future they purport to describe.
In any case, predicting the future behaviour of self-reflexive agents is not like predicting the behaviour of non-intelligent physical systems. Some predictions are self-fulfilling; other predictions are self-stultifying; and the public forecasts of politicians, social scientists, singularitarians and transhumanists should all be viewed in this light.
With this in mind, here goes….
I think the greatest underlying source of existential and global catastrophic risk lies in male human primates doing what evolution “designed” male human primates to do, namely wage war.  Unfortunately, we now have thermonuclear weapons to do so.
Bad news? I fear we’re sleepwalking towards the abyss. Some of the trillions of dollars of weaponry we’re stockpiling designed to kill and maim rival humans will be used in armed conflict between nation states. Tens of millions and possibly hundreds of millions of people may perish in thermonuclear war. Multiple possible flash-points exist. I don’t know if global catastrophe can be averted. For evolutionary reasons, male humans are biologically primed for competition and violence. Perhaps the least sociologically implausible prevention-measure would be a voluntary transfer of the monopoly of violence currently claimed by state actors to the United Nations. But I wouldn’t count on any such transfer of power this side of Armageddon.
I probably sound a naive optimist. I anticipate a future of paradise engineering. One species of recursively self-improving organic robot is poised to master its own genetic source code and bootstrap its way to full-spectrum superintelligence. The biology of suffering, aging and disease will shortly pass into history. A future discipline of compassionate biology will replace conservation biology. Our descendants will be animated by gradients of genetically preprogrammed bliss orders of magnitude richer than anything physiologically accessible today. A few centuries hence, no experience below “hedonic zero” will pollute our forward light-cone.
Freeman Dyson prophesies that soon we’ll “be writing genomes as fluently as Blake and Byron wrote verses”. If so, I’m not sure about timescales. However, “narrow” artificial intelligence and powerful gene-authoring software tools will shortly enable humans to edit our own genetic source code in accelerating cycles of recursive self-improvement. In consequence, human intelligence will be progressively amplified and enriched. Youth, vitality and lifespans will be extended indefinitely. Suffering, depression and experience below “hedonic zero” will be relegated to history. Human traits such as weakness of will, the struggle for meaning and significance, quasi-sociopathic empathy deficits, and a host of mediocre states of mind that currently pass for mental health will increasingly become optional as we bootstrap our way to posthumanity.
Not least, a growing mastery of our biological reward circuitry will allow the upper bounds of human “peak experiences” to be pushed unimaginably higher. Likewise, hedonic set-points can be genetically recalibrated. Everyday life later this century will potentially be animated by gradients of intelligent bliss.
Bioconservative critics will doubtless worry that “something valuable will be lost” when responsible prospective parents stop playing genetic roulette as the reproductive revolution of “designer babies” unfolds. Tomorrow’s parents-to-be will opt for preimplantation genetic diagnosis and “designer zygotes” to ensure invincible physical and mental health for their future children. Among young adults, novel states of consciousness as different as waking from dreaming are likely to migrate from psychedelic chemists working in the scientific counterculture to mainstream society. “Bad trips” will become physiologically impossible because their molecular signature is absent.
Unfortunately, words fail here. Post-Darwinian consciousness is likely to be incomprehensible to archaic Homo sapiens.
Ethically, I think the greatest ethical change ahead this century may be the antispeciest revolution. This global transition will probably follow rather than precede the commercialisation of gourmet in vitro meat and the end of factory farming and the death factories. It’s worth stressing that the antispeciesist doesn’t claim members of all species are of equal value. S/he argues simply that beings of equivalent sentience are of equal value. Hence they deserve to be treated accordingly – regardless of gender, race or species.
Pigs, sheep and cows are of equivalent sentience to human infants, prelinguistic toddlers, victims of Alzheimer’s disease and the severely intellectually handicapped. Only arbitrary anthropocentric bias leads us to kill, abuse and exploit the former and care for the latter. Despite superior intelligence, I suspect our grandchildren may struggle to comprehend what their grandparents did to other sentient beings.
AGI? Ben Goertzel has projected this timeline for AGI development:
2026—imposition of global AGI Nanny to ward off existential risks
2030—Singularity, managed by the AGI Nanny
Well, I’d argue [that AGI] is a form of anthropomorphic projection on our part to ascribe intelligence or mind to digital computers. Believers in digital sentience, let alone digital (super)intelligence, need to explain Moravec’s paradox. 
For sure, digital computers can be used to model everything from the weather to the Big Bang to thermonuclear reactions. Yet why is, say, a bumble bee more successful in navigating its environment in open-field contexts than the most advanced artificial robot the Pentagon can build today? The evolutionary success of biological lifeforms since the Cambrian Explosion has turned on the computational capacity of organic robots to solve the binding problem  and generate cross-morally matched, real-time simulations of the mind-independent world.
On theoretical grounds, I predict classical digital computers will never be capable of generating unitary phenomenal minds, unitary selves or unitary virtual worlds. In short, digital computers are invincibly ignorant zombies.  By their very nature, they can never “wake up” and explore the manifold varieties of sentience.
This essay was originally published by the Institute for Ethics and Emerging Technology HERE