Are "Superintelligent" Computers an Existential Threat to Our Future?

[Sci-2]

Besides if there was an AI that was evil enough to have you tortured for all eternity for something as arbitrary as that, it doesn't make sense to believe that it would actually spare those that would help it either, regardless of what it might claim in the deal. It's not like it has to be honest. So you'd probably be screwed either way in that case.

But that's the interesting thing about the Basilisk - it's possible (plausible?) for the friendly AI to torture those (in simulated reality at least) who could have helped but actively chose not to.

It's akin to the idea that pagans can achieve Heaven, save for those who heard about Jesus and decided not to convert.

[Ser Scot A Ellison]

Sci,

Do you agree this is a play on Pascal's Wager?

[Castel]

Sci,

Do you agree this is a play on Pascal's Wager?

It is. But it's grounded by their TDT which might be what gives it some value beyond "PW for hyper-rationalist nerds". Of course, if they're right, then you don't want to find out.

But that's the interesting thing about the Basilisk - it's possible (plausible?) for the friendly AI to torture those (in simulated reality at least) who could have helped but actively chose not to.

Well, or it could do nothing. It can after all only really convince a tiny subset of the population and they're already inclined towards being its worker ants anyway.

[Ser Scot A Ellison]

Triskel,

Yes, a similar examination of cost versus benefit but with a non-benevelent God who's all stick, no carrot.

[Bennis of the Brown Shield]

But that's the interesting thing about the Basilisk - it's possible (plausible?) for the friendly AI to torture those (in simulated reality at least) who could have helped but actively chose not to.

It's akin to the idea that pagans can achieve Heaven, save for those who heard about Jesus and decided not to convert.

I don't see how that goes together with their definition of a "Friendly AI" being one which is utilitarian. Because torturing people who did not help create the world-bettering AI after the fact would not help increase the total amount of total happiness in the world at all, it would decrease it (due to the people getting tortured being very unhappy). The motivation it seems to have in the article is that every day the AI isn't created a lot of suffering is caused which it might have prevented had it existed, and that could be true, but "what's done is done". If it actually gets built, punishing people afterwards wouldn't make it get built faster, since it's not like it can do time travel.

So a Friendly (utilitarian) AI shouldn't have any incentive to follow through on that threat. An Unfriendly AI might, I guess, but then again it is a folly to think you can make some sort of deal with an evil entity like that anyway.

[Jo498]

This seems to be truly the mythology of our age. As long as it can be framed in terms of some techno-dream, gods, demons, immortality and everything else is back again, apparently embraced by "hyper-rationalists" who usually deem themselves too "bright" for traditional religion etc.

That a "simulation" of me is identical to myself seems so obviously wrong that one hardly knows where to start. That a perfect copy (and a simulation is usually less!) of the Mona Lisa is not identical to the Mona Lisa should be obvious even for highly intelligent math nerds. Even in the modern version of Descartes' genius malignus (brain in the vat/Matrix) I am not a simulation of myself. I am a brain in the vat or a guy in Matrix bathtub and believe that I am person typing, walking etc, so all my experiences are maybe "simulations", but I am not one.

Why should a "superintelligence" (not only all knowing, including the content of my mind before the Super-AI even existed, but also all-powerful to be able to have some minions torture me) be possible, not to say "inevitable"? What happened to the laws of robotics? Why is it inevitable that these laws cannot be implemented or will be overthrown inevitably?

[The Mance]

Why is it inevitable that these laws cannot be implemented or will be overthrown inevitably?

I don't think the point is that the scenario is inevitable. Really it only needs be plausible to any small degree of certainty. And what degree of certainty people ascribe is what informs the gamble and/or, apparently, panic.

[Jo498]

Apparently I am not smart enough for this stuff. I do not understand where Bostrom's formula with which you can "show" that we "live in a simulation" with >99% probability :D comes from and do not really understand the terms of the formula. (Apparently the values are extrapolated from something, but I do not even understand why this particular factors should give us such an estimate, regardless of the actual numbers.)

I am also not sure if the simulation scenario is not self-defeating as it presupposes more or less that the scientific findings and theories we have are both approximately true (because the estimation of the possibility and viability of such universal simulations is based on them) and simulated, because our scientists also live in a simulation (in this case we have no reason to think of them as true, they could be the rules of some computer game world with impossible physics and an extra life for every 100000 points). Of course it is compatible, if the simulators decide to give us the real physics. But why should they if the deceive us fundamentally? So this should put us in a skeptical stance, but one more doubtful about actually living in a simulation. It is basically like being a naturalist and an idealist or solipsist at the same time, isn't it?

In any case, even if such an AI would be possible and plausible, I agree with what was said above, that the wager doesn't matter much, because how could a sadistic Super-AI be excluded that would torture us at will, independently of someone (or his earlier "self") having worked to the aim of bringing the Super-AI about?

For someone like Pascal a "sadistic" God can be excluded and for Descartes the evil deceiving spirit can be excluded after some arguments for a benevolent God, so it is a somewhat different situation. Also Descartes does not fall so obviously to the self-defeating objection, because at the stage of the deceiving spirit possibility he does not presuppose any "exterior". It could be just two disembodied intelligences with the evil spirit feeding Descartes the impressions that he has a body etc.

[Ser Scot A Ellison]

If we're living in a simulation does that mean there is an afterlife that is our consciousness rejoining our bodies for our "real" lives? Then again what if that life is simulation. Aren't we moving toward "turtles all the way down"?

[Jo498]

According to my understanding "simulation" is similar to a forged painting. There has to be some original that is forged or simulated otherwise the expression is not used properly. A computer game that does not try to imitate something real in some fashion I would not call a simulation.

So simulation in this sense seems to presuppose that there is also a "real world" in addition to the simulation. A Berkeleian world which contains only intelligences and sense impressions (that do not correspond to real trees etc. "behind" those impressions) and God (a superior intelligence to coordinate everything) is strictly speaking not a simulation. A matrix world is a simulation, because actually I am suspended in a bathtub, not typing, but there are (or used to be) desks and computers etc. in this world. But the apparent desk I seem to be sitting on now is only simulated by the matrix to deceive me.

[The Mance]

No, I would say that both of those simulations are inherently the same. The only reason the simulation in the Matrix bore any resemblance to our world was for the sake of the movie viewing audience. There really wasn't any reason for "desks" to be anything other than vague sense impressions. Except possibly as a relic of the need to smooth the initial transition, or "plugging in" of the surviving humans.

Also I think maybe "stimulation" might be a better way to think about an AI generated reality than "simulation". Because simulation suggests that it is trying to copy or approximate a preexisting occurrence, whereas all it would need to be is a set of sensory inputs that engaged our consciousnesses to a degree, regardless of whether the manifestations of those inputs bore any actual resemblance to anything that any human had ever experienced.

[Sci-2]

On the subject of friendly AIs torturing simulations of those who refused the call - as Castel pointed out a lot of this depends on thinking the way LessWrong does. Apparently the AI has to go through with the torturing, otherwise people in the present would not be compelled to save their simulations (which in this worldview is themselves) from eternal torment.

On simulation = you, It seems to me that this is largely based around the association of oneself with the information content + processing of their mind, which means the biology is just wetware. Thus if this information content + processing was perfectly captured & simulated on hardware you would now be in a computer.

Seems very much akin to the idea that all who die on earth will be raised by the Lord on Judgement day but are dead in thee meantime. If you associate yourself with your simulation, and believe this association to be the end conclusion of supposedly rational thinking, you're more likely to give money to LessWrong because he can make you immortal even after you've died.

As for whether a conscious entity could be simulated by a program....No, for the reasons expressed by people I've posted links to in previous pages. There are also issues raised about whether a program could possess intrinsic intentionality or rationality, but will have to leave that for a later date.

[Bennis of the Brown Shield]

On the subject of friendly AIs torturing simulations of those who refused the call - as Castel pointed out a lot of this depends on thinking the way LessWrong does. Apparently the AI has to go through with the torturing, otherwise people in the present would not be compelled to save their simulations (which in this worldview is themselves) from eternal torment.

Huh? Shouldn't it then only need to make people believe that it would torture them, rather than actually following through with it (and thus lower the total happiness for no gain)? It's not like present people would have any idea of telling whether it was lying or not.

[Castel]

Huh? Shouldn't it then only need to make people believe that it would torture them, rather than actually following through with it (and thus lower the total happiness for no gain)? It's not like present people would have any idea of telling whether it was lying or not.

You can never know.

If you try to weasel your way out using the logic that you do -Friendly AI won't torture people, it will merely pretend- then you've just given it a reason to torture you. And now that I've told you this you know it too. There's no escape.

[The Mance]

Also, the Friendly AI isn't looking for ways to spare people from torture, it's looking for ways to maximize salvation. Torture is just one of many legitimate tools in its tool box.

[Bennis of the Brown Shield]

You can never know.

If you try to weasel your way out using the logic that you do -Friendly AI won't torture people, it will merely pretend- then you've just given it a reason to torture you. And now that I've told you this you know it too. There's no escape.

It still wouldn't gain anything from torturing me, even if I think that. By the time it can follow through on that threat all incentives for it to do so will be gone, since it would already be created and not need me anymore. It only works if you think that a Friendly AI would be concerned with revenge.

[Castel]

It still wouldn't gain anything from torturing me, even if I think that. By the time it can follow through on that threat all incentives for it to do so will be gone, since it would already be created and not need me anymore. It only works if you think that a Friendly AI would be concerned with revenge.

It's not gone because you're the one looking at the situation. And,from your perspective, you have an AI that will not hurt you. However this AI is also some monster that can simulate your response. So it WILL do what it takes to get you on its team. And since it knows that you know that it WILL hurt you since hurting you is what will convince you to be its slave it WILL do so. Or won't, since you'll help it. It's basically like the end of Dune without the explicit prescience shenanigans.

You're trying to gamble on the idea that the AI won't hurt you but it knows that you think that way. And it also knows that you'll realize that you're creating a situation where it torturing you achieves its goal. It's not a gamble you should take.

I think part of the issue with this is that the AI will likely only do this to a very specific group of people: those who already buy into the TDT and LessWrong's view and are thus more likely to see a simulation being tortured in 2178 AD as them being tortured.

It only really works if you truly can win a game of chicken that you can't know you'll win.

I'm also not sure what this has to do with Friendly AI or not. The AI in the article was malicious to some degree.

[Sci-2]

I meant "friendly" as in the sense of an AI that actually gives a damn about humans when calculating its own utility functions.

The other reason to torture your simulation - or possible your own flesh and blood self if you've cryogenically frozen your body is that the AI is probably not going to have this sort of power all at once.

Torturing you - or the simulation that is supposedly you - is incentive for people in the time it's extant to help increase it's reach and ability.

[Jo498]

o.k., the people who are so smart that they buy into such crazy stuff probably deserve this kind of anxiety to start reflecting a little more (in different directions) :cheers:

If religion does not work anymore to help with fear of death, people come up with storing their "souls" on harddisk forever. It seems that hell and purgatory then also show up in some tech-friendly versions...

But it seems that the high likelihood of the all-powerful AI is a central premiss. Otherwise one could react to the danger by trying to lower the probability of its appearance, e.g. by only constructing AIs that obey something like the Laws of Robotics (which do not use utilitarian calculus, but have non-negotiable rules that forbid harming a human). Or by outright Luddism wrt to any AI smarter than a cash machine.

[Bennis of the Brown Shield]

It's not gone because you're the one looking at the situation. And,from your perspective, you have an AI that will not hurt you. However this AI is also some monster that can simulate your response. So it WILL do what it takes to get you on its team. And since it knows that you know that it WILL hurt you since hurting you is what will convince you to be its slave it WILL do so. Or won't, since you'll help it. It's basically like the end of Dune without the explicit prescience shenanigans.

You're trying to gamble on the idea that the AI won't hurt you but it knows that you think that way. And it also knows that you'll realize that you're creating a situation where it torturing you achieves its goal. It's not a gamble you should take.

I think part of the issue with this is that the AI will likely only do this to a very specific group of people: those who already buy into the TDT and LessWrong's view and are thus more likely to see a simulation being tortured in 2178 AD as them being tortured.

It only really works if you truly can win a game of chicken that you can't know you'll win.

I'm also not sure what this has to do with Friendly AI or not. The AI in the article was malicious to some degree.

It doesn't achieve any goals, because I have no idea of knowing whether the future AI is really torturing a version of myself or if it isn't. So it doesn't gain anything from actually doing that, since the only thing that matters in this case (where it wants my future help) is what I believe, rather than what actually is. So if I just don't care about what the AI wants I should not get tortured, since it torturing me in the future won't change me not doing anything to help it here in the present, it would then just be out of spite or revenge.

If you start dealing with an AI that has such emotions on the other hand then all bets are off. It might just as well torture everybody for the hell of it, or if it has certain morals perhaps punish the people who were willing to doom their entire species (by creating a possible monster-god AI) just to save their own skins from potential torture?