I often think about who I want to be. I want to make the world better. I want to live with someone I love. I want a community of friends around me. Dreaming about this helps me figure out the steps to get there.
On a larger scale, I think about heaven. Where should society end up? What is the picture we are aiming towards? Is it white clouds and eternal joy? Is it a near infinite amount of happy shrimp? What do we want?
The good future should be:
Universal
I think it's not the good future unless it's for everyone. I'm not saying it's equally good. But the way it's good has a baseline for everyone. If they don't, I'm not sure I want to be in this future, and if I'm alive and don't want to be in it (or have struggled and died to stop it) it doesn't sound like the good future
I can imagine that the good future is briefly bad for some, but I hope it is in a way I'd endorse if I were them and I'm pretty wary regardless.
Stable
The good future has to not involve conflict over the future's future. If it does, then at the back of many minds will be the fear that actually heaven is about to come to an end. At that point you want to turn off your heavenly hot tub and start some zero sum games. This, also, does not sound like heaven.
Good
I sort of haven't included this as part of the other two, so lets be clear. The good future is really good. Whether you are a bug or a human or a AI running on the energy output of an entire sun, it's good for you. You sense it's going pretty well. You are glad to be alive. There isn't confusing badness.
I note that I am avoiding exactly how good and how this trades against universality. We'll sort of get to that.
Growing
My personal heaven involves growth. It involves becoming more, understanding more. Enlightenment too cheap to meter. I'm not against wire-heading per se, but I don't get why I'd want to while there are so many systems to understand and people to chat to.
I think I'd be pretty wary of a future where we get somewhere and stop.
Conflicts
There are, I think some issues with the things I've laid out.
Good vs Universal. What if it's really good for a few but bad for many? What about beings who might exist? This is well trodden ground, I lean slightly in favour of more beings, even if average happiness is lower (as anyone who has come to one of my parties knows), but I could be persuaded. I am pretty wary of thinking I know better than others, especially others who are less powerful (as the non-yet-alive often are).
I am also pretty in favour of consent and property rights. The
Repugnant Conclusion can be a great end state whilst also not being a path we can get to - maybe those who are currently alive don't want to give up their resources for new beings. And maybe the governance process doesn't force them to. Who am I to disagree? Sometimes forcing good outcomes can be worse than merely mediocre ones.
Growing vs Stable. How do you have stable growth? If the future can change then can't their be winners and losers? Can't there be those who want to see the future be more Randian and less luxury space communism? I hope that we end up good enough at coordinating that those who create lots of value are often passing it back to mega-foundations with the good of beings at their centre. But this doesn't seem axiomatic.
Growing vs Good. If change is unpredictable, it might be bad.
When I left Christianity, how could I know that my life after was going to be good. Likewise how can we know we are getting better futures without some chance that we get worse ones. I guess it's about equilibria - changes that are pretty stable unless everyone is very confident, but still.
Finally then some critiques of common pretty good heavens and some discussions of some disconcertingly plausible midtopias. I'm not forecasting here, please do not consider these likely outcomes.
The Culture. The culture is a universe where AI minds run the pan-human milieu. It is transhuman Communism with a edgey.. edge. But there is a clear issue. Why wouldn't you become a mind? This essay covers it in more detail, but yeah, why as a human wouldn't you become a ship-mind (AI) capable of actually engaging with the big problems of the universe. And ships also goof around and have bodies sometimes too. I mean, why be human when minor-godhood is on offer.
CS Lewis. Frankly I think CS Lewis's heavens, which I generally enjoy (eg
The Great Divorce) suffer a pretty similar problem. I don't believe God is real and I am not sure I would trust a being a random arbitrarily powerful being. If God existed, I'd want to be him, or perhaps be so utterly outmatched that I had no choice (which is kind of an interesting theology now that I think about it). Suffice it to say I don't think that's the world we are in, so while I want an every increasing mountain of joy and discovery for all beings, I don't know who is going to create and maintain it.
The good AI. There is one all powerful AI, but luckily, it's good. Phew. This doesn't seem like a good thing to hope for because.. what if it's not the good AI? If this goes well, it really is the good future. But it feels to me that near this good future are 1000x as many bad futures, where the AI isn't good, or someone scared of being ruled (eg me) screws it up for everyone. Unlikeliness is a bad thing in terms of heaven.
In summary, I want there to be growth for me, which likely involves being genuinely powerful, but if I am powerful then so are others, so this may not be stable. And if only one person is powerful then that sounds like a dictatorship, which is probably bad.
And so, some midtopias.
Gay Mediocre AI Capitalism. The future is still cool and queer, but we exist in the gaps between a constantly expanding swarm of AI minds. Maybe we can become them, maybe not. You have your own continent, but AIs control 90% of everything and they are gonna charge you to mail your mum. There are ads everywhere, but for the best book you've ever read, and the match of your dreams and ultrabeer. It's great, it's terrible, welcome to the future.
Ems. Robin Hanson wrote the book on human emulation. The long and short of it is, that even if you choose to play your harp on a cloud, someone else is gonna make a bajillion copies of themselves and make moves on all of the energy in the universe. There is sort of a black market on growth, which is likely to be hard to suppress, leading to most of the universe being run by copies the most competent and coordinated, but least scrupulous people. Because if it weren't, then someone more competent and coordinated and less scrupulous would break laws and do it.
Regulationworld. Maybe all the really broken technologies are banned for some reason and we plod along forever forever with things improving a bit. This seems nice, but also, much less good than possible and very unstable. As soon as ems or AGI gets built, we are back into the frying pan.
This article isn't really about a solutions, but to lay out the problem as I understand it. What should we aim for? How do we coordinate around a future we all think is great? Please comment with things I should have read. I'm sure philosophers have written about this and there are some good novels too. What is a minimum viable heaven? How do we built it?