mccHave you ever noticed how many canonical "paradoxes" just sort of evaporate if you decline to recognize Bayesian inference as a thing that works
Hmm so it looks like you started with some absurd priors and you were able to use them to prove some absurd conclusions. Now you're acting like this is a fundamental challenge to the idea of "rationality" and you've made a wikipedia page. Seems to me like you just selected some absurd priors. At absolute most what you've proven is that game theory kind of sucks
(This might be kind of vague so this is the kind of thing I'm talking about: https://en.wikipedia.org/wiki/Pascal%27s_mugging A shocking number of problems of this type that make me immediately respond with "why do you think this is a difficult problem?" seem to wind up mentioning Eliezer Yudkowsky when you look into why people are talking about them.)
sanaINTERNET RATIONALIST: Consider the following thought experiment. Imagine a hyperintelligent artificial intelligence–
ME: No
INTERNET RATIONALIST: What
ME: I am declining to imagine the hyperintelligent artificial intelligence.
INTERNET RATIONALIST:
ME: I'm thinking about birds right now
INTERNET RATIONALIST:
ME: Dozens of crows, perched atop great standing stones
Jonas Wisser@mcc You know, I already knew Roko's Basilisk was stupid, but for some reason it never occurred to me before now that it's just self-proclaimed rationalists reinventing God and Hell the hard way.
@jwisser The way I first learned about Bayesian reasoning was in the evolution-vs-intelligent-design-arguments Usenet group. Most of the laziest proofs of the existence of God by internet theists leveraged Bayes' theorem, and could be most easily punctured with the sentence "you selected bad priors". Now a couple decades pass and people who know about Bayes' theorem but not theology are re-inventing "God" from first principles for different reasons, but with very similar bad priors
noop_noob
DataShade@noop_noob @mcc @jwisser Sadly, I've never been able to stay awake through any explanation of how that core belief was established aside from selecting remarkably bad priors.
NinjaDebuggerNo. It was a trap to point out a giant hole in the "consistent rational framework" a bunch of people were trying to use. No more, no less.
The fact that it's literally Pascal's Wager for techbros just makes it more hilarious.
delong
Rob Hafner 
Tony Wells
Jesse Morris
Irenes (many)
Chris Silverman 🌻@mcc ME: why yes, I am the hyperintelligent artificial intelligence
INTERNET RATIONALIST: Um,
vulp@mcc personally I would solve AI alignment by asking the malicious genie for three extra wishes
Jacob Harris@mcc I reached this point with the whole stupid “nuclear bomb that only disarms if you recite a racial slur” thought experiment a few months back. I decline to be part of imagining that is ever plausible or the basis of a serious argument.
Brittle Anxious Nonlinear Inco@harrisj @mcc I wish I had a nuke like that, I would just stand there, next to it, not saying slurs and fighting anyone trying to get to it to say slurs, with my fists, wait, can I have stealth multirole combat aircraft for this one? Heck, I would bring the nuke with me on the trolley, that'd show those trolley people whose boss.
YetAnotherGeekGuy 
Sarah@harrisj
LOL. What a ridiculous thought experiment!
A) never will happen and B) even if it did happen, it would have no bearing on any situation in which people actually claim the "right" to speak slurs.
Like, yes, I guess in this alternate universe you're hypothesizing, I'd speak a racial slur if it was the magic key to disarm a bomb about to kill millions of people. What on Earth has that got to do with ANYTHING that has or will happen anywhere ever?
@SarahAnneDipity @harrisj The scenario was invented by people who want to argue it's okay to say slurs, so they invented a hypothetical where it's morally necessary to say slurs just so they can win an argument about saying slurs. So actually it's very straightforward, it's just dishonest.
And since this is the internet, while I think it's clear that I understood this was not your thought experiment I worry because I hate when someone comes into my mentions appearing to be arguing with ME about something *I* was critiquing.
That was just a new one for me and really made me laugh.
RAOF@mcc This is going to expose me as someone that's spent too much time looking at that stuff, but I particularly like the one that's “What if there was this magical superintelligence that leaves people with two boxes, one of which always contains $10 the other of which reliably, repeatedly, and observably contains either $1,000,000 if you don't open the first box but nothing if you do open the first box” and then has a huge convoluted philosophical argument trying to work out how to make “only open the box you know will contain $1,000,000” the “rational” choice.
Instead of the rather more obvious argument “this thing observably happens, therefore my assumption that it cannot is incorrect”.
@RAOF or even the contrapositive (? Did I use that word right) of that argument, "the magic box literally does not exist in reality, because this is a thought experiment, therefore I can make the argument it cannot exist"
@mcc That is another perfectly sensible option!
Either you work within the logic of the hypothetical, where the magic box demonstrably exists, and so working out how to rationalise the behaviour of the magic box existing isn't very interesting, or you reject the magic box and the whole thing is void.
Alex Zani@AlexandreZani @RAOF Well… no, I think I'd argue the supernatural powers *are* necessary, because if the predictor is like a really good friend then suddenly I have to start asking questions like, *is* there any person on earth who I've revealed enough of myself to they can predict how I'd behave in extreme situations, and suddenly I'm judging against "how well do they know me" and not the probabilities the thought experiment is supposed to be about.
@AlexandreZani @RAOF And if the assigning the weights on the inputs to the probability function that comprises the thought experiment turns out to be a harder problem than executing the logic of the probability function itself, then… isn't what the thought experiment has ultimately shown, is that the probability function isn't useful?
Because that was my point to start with– if we're allowed to bring "this entire methodology seems to be working kind of poorly" in as a possibility…
@AlexandreZani @RAOF …Well, then some of the hard parts get easy!
@mcc @RAOF I think we have to look at the purpose of Newcomb's problem to evaluate it. It's meant to be thrown at the two main decision theories philosophers argue about. Those theories are usually meant to address all situations. Maybe you can't be Newcombed. But I promise you the people who argue endlessly about those decision theories can be put in that situation. So if those theories are total, they have to address it...
@AlexandreZani @RAOF Well, I don't think it's a challenge to "rationality". But I can identify specific self-identified rationalists who seem to think that adopting rationality as a personal philosophy means needing to grapple with this problem some way or another. Maybe if I'd read their Harry Potter fanfic I'd know enough about how they got there to extend this with "…because they fundamentally incorporated dubious approaches to decision theory into their construction of rationalism", idk.
Avram Grumer
Sparkpaws™ 5000®Wait, is this thread discussing rationalism or Christian apologetics? [They are that similar, though religion at least has rituals that many find comforting long after they've grown to find the philosophy repulsive]
The rightful place for deductive logic is subservient to empirical observation and inductive reasoning.
серафими многоꙮчитїи@mcc A lot of the issue is also to do with being deliberately sloppy about 'possible' like, do you mean modality, and if so which one? Or is it some kind of quantification, and if so what are the details? Or is it a simple predicate? - eg doing moral reasoning that treats an imagined 10^80 possible intelligences in cyberheaven as real actual present existing people, which is nonsense.
@mcc They have to selectively move between meanings of 'possible' at different parts of their argument and their deliberate sloppiness (and logically invalid moves) are disguised with a heavy layer of the aesthetic of logic.
JP@flaviusb @mcc "aesthetic of logic" is a good way to put it because so much of that world is about performing intelligence, believing in and trying to be superhumanly smart kids who alone can stand against the invented enemy, even-superer-humanly smart computers. just a weird sad intellectual limb to have climbed out onto.
Jesse@jplebreton @flaviusb @mcc Tema Okun could have spared herself and the left nonprofit world a lot of trouble by replacing most of her pamphlet on white supremacy culture with a big arrow pointing to Less Wrong.
Ray “Tracing” Chen@jplebreton @flaviusb @mcc I saw a tiktok of a dog who visits a bush every day on its walk because somebody had thrown out a lasagna there once. it sounds fancier if you replace the dog with “Pascal” and a lasagna with “a million trillion dollars” or “simulated AI heaven”. the dog is still more reasonable because it saw the lasagna with its own eyes
Ryusui@theryusui @flaviusb ok so you say this but the first season of Discovery actually had a subplot where a group of "logic extremists" logicked themselves into being a Vulkan alt-right and started assassinating people.
(and… I guess probably Spock would have slapped them, but he didn't get cast until season 2! So instead Spock's sister had to do it…)
@mcc @theryusui I mean, 'Mr Spock' *was* the canonical example that my logic professors used to use of the aesthetics of logic disguising deliberate sloppiness and logically invalid moves.
🍁Maple🍁An old man in a white traveling cloak with vermillion tunic carries a pole over one shoulder to which is tied sheaves of grain. In his free hand is a little iron sickle, and as he walks along a dirt path, two little foxes prance about his heels. One black and one white. One with a key and the other with a gourd full of wine.
Matthew Green@mcc You are the first person in history who got a rationalist to stop talking.
@matthew_d_green Note it was not a real rationalist, but a hypothetical rationalist I posited for the sake of a thought experiment, but then again as I understand effective altruists believe hypothetical people posited for thought experiments are individually as valid as real ones so whatever
@mcc The more hypothetical people you create the better the utilitarian calculus, so you’re doing good.
ZenithAstralis@mcc me: just having read a book involving pairs of crows who together could very convincingly mimic intelligent thought and speech, yet who were very insistent that they (and everyone else) were in fact just acting intelligent, while not actually being aware whatsoever.
me: "so what if all these crows could talk, and process vast quantities of information, BUT were only interested in shiny objects and bits of food."
me: "so what if all these crows could talk, and process vast quantities of information, BUT were only interested in shiny objects and bits of food."
Chris [list of emoji]So I read^Wskimmed the Wikipedia page and it seems like if you round any probability below (say) 10**-9 to zero, you stop making these sorts of stupid decisions.
Toby Haynes@TobyHaynes Cute :3
aeva@mcc it's a shame that shouting "THIS STATEMENT IS FALSE" doesn't really work to ward off hyper-intelligent something anothers and also armchair logicians
eevee 🦊@mcc this does suggest something about how your hypothetical rationalist views their own imagination. perhaps it's limited enough that they only imagine things that are plausible, and as a result they've concluded that anything they can imagine is plausible
Blake C. StaceySuperintelligent AI inside a box: [lengthy argument about why the listener must release it from the box]
Bartleby the Scrivener: I would prefer not to
and… scene
Benjamin Johns@mcc Let me just grab my handy notebook of hypothetical scenarios I'm willing to contemplate.
Falcon Darkstar@mcc appreciating the example of a superhuman AGI as the impossible assumption.
"While we are assuming things that don't exist, that we have no reason to believe will ever exist, why don't we assume Mars is a short bus ride away and has breathable air? Then, obviously, colonizing it will ease the suffering of billions."
Johan Halse@mcc it’s a great hack to use against mathematicians too! When they say things like “Let p be a complex number” you can just say no. Checkmate
Panic In The Darkroom