Wiki Commentary On Who Are Singularitarians

• <observer> So far your essay is very good. I haven't noticed any problems that you aren't likely to notice when you do your next revision.
• <MichaelA> ahh, which problems
• <MichaelA> I've looked over this thing like 20 times
• <observer> minor typos, simple but important details left out
• <MichaelA> hah, which simple but important details

• <observer> "All of humanity's problems can be attributed to our lack of problem-solving ability."
• <observer> what about selfishness and apathy?
• <MichaelA> those are problems
• <MichaelA> that we lack the ability to solve
• <MichaelA> hm, I see what you're saying
• <observer> perhaps the best way to fix that would be with a footnote.
• <MichaelA> yup

• <observer> "Preliminary projections of the capabilities of nanomachinery from known physical principles suggest that their impact will be greater than several Industrial Revolutions packed into a single decade."
• <observer> how do you measure this?
• <MichaelA> hard to measure quantitatively, I would guess Kurzweilesque type statistics

• <observer> minor typo: "nanomachiggtnery"
• <MichaelA> I used that phrase because Chris Phoenix has, actually
• <MichaelA> heh
• <MichaelA> nanomachiggtnery
• <MichaelA> sounds like something an SI would invent :D

• <observer> there was another spot before that where the wording was a bit awkward, but I forget where.

• <observer> the essay seems like an unnecessarily long answer to the question "Who are Singularitarians?", but most of these details are actually required to give an accurate answer.
• <MichaelA> it was originally *much* shorter
• <MichaelA> until I realized that only people who were already Singularitarians would probably agree with it
• <Nick Hay> are the important things said first?
• <Nick Hay> ie. if people get bored and stop reading :)
• <MichaelA> heh, yes
• <MichaelA> they are
• <MichaelA> I even added another paragraph at the beginning just this morning
• <MichaelA> about smartness
• <MichaelA> Who Are Singularitarians

• <observer> also, I was wrong about the essay needing to have lots of links to the lexicon

• <observer> "an AI might take steps to understand the eternal world"
• <observer> should be "external", I assume

• <observer> "In fact, imagining anything specific will never serve as much more than a lower bound on the capabilities of a truly transhuman AI."
• <observer> does ontotechnology count as "anything specific"?
• <observer> ontotechnology is *the* upper bound
• <MichaelA> ontotechnology is incredibly general
• <MichaelA> by "specific" that would mean specific methods and approaches, etc
• <MichaelA> perhaps a wording issue, though
• <Nick Hay> ontotechnology is one of Eliezer's errors
• <Nick Hay> at least in the sense he meant it
• <Nick Hay> according to him
• <MichaelA> hm, I wonder if he really considers it an "error" or whether he just thinks it is of too little value to be worth paying attention to, and therefore an error to create the meme in the first place
• <observer> perhaps this would be another good place for a footnote.
• <Nick Hay> he mentioned the idea of ontotechnology itself was erroneous
• <Nick Hay> he orginally intended it as a way to engineer an objective morality into the universe, and even more erroneous goal
• <MichaelA> heh
• <MichaelA> I had no idea that is how he originally intended it
• <MichaelA> but that's pretty funny to hear
• <Nick Hay> probably something to do with trying to step outside of something you can't step outside of
• <MichaelA> ah
• <MichaelA> I still don't see how like, messing with fundamental physical rules wouldn't be considered "ontotechnology"
• <MichaelA> although I guess that wouldn't make them fundamental anymore
• <Nick Hay> aye
• <MichaelA> this just sounds like semantics, as usual
• <Nick Hay> something like that
• <Nick Hay> quite possibly
• <Nick Hay> but given Eliezer's goals, ontotechnology probably had extra meaning
• <observer> The TransTropians are using two of Eliezer's most dangerous essays as part of their core philosophy: PtS and the "educated guesses" essay
• <MichaelA> yeah, I told Eliezer about that
• <MichaelA> I'll email Dalibor and ask him to please take them down
• <observer> what's worse is that they included the disclaimer that they don't believe in any of the stuff about altruism or Friendly AI
• <Nick Hay> not just "the most fundamental and complete ability to change things"
• <MichaelA> I was given the offer to rewrite Transtopia's section on Singularitarianism, would that be worth doing?
• <MichaelA> I want to do it but I have little idea what I write
• <observer> heh, isn't that what this essay is for?
• <observer> or do you need something much shorter?
• <MichaelA> Who Are Singularitarians? no, that's not supposed to be for Transtopia :)
• <MichaelA> shorter and faaar more neutral
• <observer> perhaps you could write a short version full of links to the longer version
• <observer> since when is transtropia neutral?
• <MichaelA> Dalibor apparently is allowing me to do it, so they (he) may be coming more neutral

• <observer> "A problem that was "impossible" to the engineer of the 18th century has probably been solved, oversolved, improved or even replaced by the knowledge of the engineer of the 21st century."
• <observer> this could be changed to "many problems"
• <MichaelA> ah, mmkay

• <observer> "Selfishness sometimes goes hand in hand with power, and vice versa, but we have little reason to believe that this phenomenon holds outside of humans."
• <observer> should be "outside of evolved intelligences", or something like that
• <MichaelA> ah, okay

• <observer> "What would an intelligence care about if it were selfless? Ideally, it would care about fulfilling the volitional desires of all sentient beings equally, but could theoretically care about anything"
• <observer> the claim that volitional morality is the "ideal" morality is still very controversial
• <Nick Hay> not sure it'd apply to *all* evolved intelligences..
• <MichaelA> yeah, I actually think "humans" is fine for that sentence
• <observer> ok
• <MichaelA> okay, I'll consider a rephrase of the volitional-morality-is-best thing
• <observer> that might be difficult

• <observer> "creating a galaxy-sized supercomputer for compiling trillions of images of happy humans"
• <observer> only trillions?
• <MichaelA> "trillions" works fine for the purpose of this essay, I think
• <MichaelA> but I could replace it with "bajillions"
• <MichaelA> which might be better
• <observer> yeah, that might also be better than "countless"
• <Nick Hay> heh, bajillions sounds silly to me
• <MichaelA> zillions?
• <MichaelA> the real number would probably be difficult to write out ;)
• <bghfhfg> ***!!!Not a real number!!!***
• <MichaelA> it has the general connotation of "really really big", and everyone knows this
• <observer> so does "countless"
• <bghfhfg> Just include a link to Staring Into the Singularity, Graham's # section.
• <MichaelA> ah, but that has the slight connotation of infinity, which gives it a negative quality
• <MichaelA> Graham's # is too big, of course!
• <MichaelA> how many images of happy humans could fit on a galaxy-sized supercomputer, I wonder
• <MichaelA> with optimal computronium, Graham's # might be possible
• <Nick Hay> bajillions sounds large, but it also sounds kind of amaturey. how about trillians, with a footnote? :)
• <bghfhfg> How about: "creating a galaxy-sized supercomputer for compiling googles (10^100 = google) of images of happy humans"
• <Nick Hay> "Technically speaking, ..."
• <MichaelA> how about trillions, with a footnote
• <observer> ok
• <Nick Hay> even better!
• <bghfhfg> Nobody reads the footnotes, especially when they're way down at the bottom.
• <observer> the end of that paragraph focuses on "fulfilling desires", though I'm not sure what a better alternative would be.
• <observer> or at least noone reads footnotes when they're not clickable
• <Nick Hay> got to be clickable
• <MichaelA> yeah, I would make it clickable on the page
• <MichaelA> you guys also missed out on a lot of italics and links
• <bghfhfg> Teachers overrate footnotes.
• <MichaelA> they do
• <MichaelA> I rarely use them
• <Nick Hay> or, you could use brackets
• <MichaelA> parentheses?
• <Nick Hay> but kind of distracting from the point
• <Nick Hay> yah
• <MichaelA> would be quite
• <bghfhfg> Or, you could not explain anything and just reference people to Google.
• <bghfhfg> Google not to be confused with a google.
• <MichaelA> you misspelled googol earlier, btw
• <MichaelA> it's googol
• <Nick Hay> googol sounds too large in any case
• <MichaelA> it does
• <MichaelA> ~10^82 atoms in the universe
• <Nick Hay> *observable
• <bghfhfg> How about a 'trillion trillions trillions trillions'?

• <observer> "Not the strawmen altruists that Objectivists talk about"
• <observer> so you're expecting your target audience to know about "strawmen" and "Objectivists"?
• <MichaelA> I linked it there
• <MichaelA> to this: http://www.objectivistcenter.org/articles/dkelley_two-strains-altruism.asp
• <MichaelA> to make it clearer
• <observer> oh
• <MichaelA> strawmen, yes
• <observer> you might want to find a link for that too
• <observer> just in case
• <MichaelA> maybe people poor at English might have trouble understanding it
• <bghfhfg> Bomb the blighters! Batten the hatches! Break out the fancy terminology!
• <MichaelA> heh
• <MichaelA> terminology can get radically more fancy than "strawmen" ;)
• <bghfhfg> Example....
• <bghfhfg> Sounds simple enough. Include a link to Google explaining what an 'altruist' is.
• <MichaelA> from 2001: "A monopolic, chromodynamic, van de Broeck warp, or gluonic processing core emerging in a type M star system would cause a cosmic event never before seen anywhere in the universe -a continuously self-organizing, sun-absorbing bush-robotic mobile supercomputer!"
• <MichaelA> actually, jeezus, that was from 2002
• <MichaelA> http://www.bjklein.com/sing/forum/topic.asp?TOPIC_ID=510
• <Nick Hay> heh
• <Nick Hay> you've improved muchly
• <MichaelA> thankies
• <MichaelA> that line was in jest, though
• <MichaelA> a jest that few readers would get, making it somewhat useless

• <observer> "Not the strawmen altruists that Objectivists talk about - genuine altruists who really pay attention to what people want, maintaining empathy and the desire to maximize freedom for everyone regardless of what opportunities for power present themselves."
• <observer> again, the desirability of "maximizing freedom" is controversial
• <observer> you would need a big footnote for that one
• <MichaelA> observer, agreed on that part
• <MichaelA> I feel sort of weird when I make claims like that because I know they'll be controversial
• <MichaelA> should probably soften the wording somewhat
• <observer> the "altruism" entry in the SL4 Lexicon is on the list of entries that need revising.
• <MichaelA> I see
• <bghfhfg> Warning: Memes spread exponentially, until they either encounter 1), competing, contradictory memes, or 2), the limit of the world's population.
• <MichaelA> they don't necessarily, they only spread exponentially if enough people tell their friends about it
• <observer> a lot of people think of altruism as abandoning all selfish desires, even if doing so doesn't actually help anyone.
• <MichaelA> hm, interesting point
• <MichaelA> unfortunate
• <bghfhfg> Since the technophobism memes aren't that widespread, that means the only limiting factor is how many people can understand it reasonably well.
• <MichaelA> I think that Blank Slate type memes are actually the main contradictory memes
• <bghfhfg> Sure they do. Even if people, on average, only tell 0.00001 friends, that's still an exponential growth. Plot the numbers on a chart.
• <MichaelA> it's not about loving or hating technology, but what you do or don't imagine as possible in the area of mind-manipulation/emulation/improvement
• <bghfhfg> What do you mean? How is the idea of 'tablua rasa', as stupid as it is, Singularity-contradictory?
• <MichaelA> not if they die before telling anybody
• <MichaelA> I'm just saying that very slow exponential growth looks the same as linear
• <bghfhfg> True. Until the meme reaches critical mass, and *boom*.
• <MichaelA> when you view the mind as a homogeneous lump of Ur-pattern, it seems incredible to create AI or improve the human brain, and the idea of "smartness" doesn't sink in either
• <bghfhfg> How so?
• <MichaelA> when you view the mind as a physical system composed of interacting parts, it seems less far-out to try to duplicate it
• <MichaelA> or augment it
• <MichaelA> because then you understand it slightly better, and can more accurately imagine what could or could not be done
• <Nick Hay> MichaelA: if the mind is a simple Blank Slate, means it's easier to copy
• <Nick Hay> maybe if you double the computing power you get something smarter
• <bghfhfg> What is the mind made of, then? Copies of my doggies?
• <Nick Hay> although improvements are far more clear with a more accurate view of the human mind
• <MichaelA> I'm just saying that Blank Slate/SSSM ideas can and do contradict Singularity-related memes at many points
• <MichaelA> that's what you would think Nick, but it doesn't actually work out like that
• <Nick Hay> the ideas aren't necessarily inconsistent, but given other background memes it doesn't seem to work
• <MichaelA> that is probably the case, yeah
• <MichaelA> oddly, Blank Slate seems to lead to an overestimation of the software complexity of the human brain (because it has to encapsulate all of who we are as a person, and that is supposed to be really complex), plus assorted anthropomorphism
• <MichaelA> instead of a literal Blank Slate, I think people imagine a really really complex "clean brain" totally tuned to pick up information of the environment to characterize itself with

• <observer> minor grammar problem: "If you believe that neither AIs, human beings, neurologically or cybernetically modified human beings could behave compassionately..."
• <MichaelA> what is the grammar problem exactly, aside from awkwardness
• <observer> that sentence needs one more "or", and it belongs in a rather awkward place
• <observer> also, that sentence is awkwardly long.
• <MichaelA> observer, I was worried about that one, thanks

• <observer> now that I see the context of that sentence about "moral systems", it does seem more awkward.
• <observer> there should be a better term than "moral system"
• <MichaelA> "Goal System"?
• <MichaelA> aside from that phrase, how else is it awkward? too long/confusing?
• <bghfhfg> Anthro_____phism? Don't ve sayzing zat dirty vurd!
• <MichaelA> is 'anthropocentrism' okay? :)
• <bghfhfg> It vas ze joke!
• <MichaelA> I know I know :p
• <MichaelA> they are both dirty words, really
• <observer> if you can't find a better phrase, another alternative is to replace "Such a moral system" with "Such an AI's moral reasoning system"

• <observer> "Fundamentally, the issue is about trust, and an AI suitable for entering Recursive Self-Improvement will need to be so trustworthy that the experts feel thoroughly comfortable with it"
• <observer> it's not about how the experts "feel", it's about how reliable the AI really is.
• <MichaelA> true
• <MichaelA> my guess is just that the experts will be our best guess at the time
• <observer> but still, it's not "fundamentally about how the experts feel"
• <observer> (that's not a direct quote, but it's close)
• <bghfhfg> Just be wary of self-proclaimed experts.
• <MichaelA> true, I shouldn't use the word "fundamentally" there
• <MichaelA> fundamentally, the issue *is* about actual trustworthiness
• <MichaelA> trustworthiness can only be assessed by humans in an age where humans are the only intelligences around, but I know how it gives the wrong impression in that context

• <observer> "One day it will be possible to custom-design our own emotions and Goal Systems, or design AIs from scratch with biologically impossible Goal Systems"
• <observer> again you're using the word "will"
• <MichaelA> ah, d'oh
• <MichaelA> will yank that out
• <MichaelA> ta ta for now

|*<-- *<MichaelA> has left sl4.org (Quit: 2004... running behind schedule, aren't we?) -->| Starglider (Starglider@dial-62-64-217-206.access.uk.tiscali.com) has joined #accelerating

• <observer> heh, you joined immediately after Michael left.
• <Starglider> This is to maintain the 'at least one Michael on the channel at all times' rule.

Other parts to consider revising:

• "if the AI actually wants to be a just person"
  • ("just person?" also, the idea of justice is typically linked to the idea of punishment)

• "A benevolent transhuman mind wouldn't decide to create a successor that is a risk to humanity because such a mind would love humanity (and sentiency in general) and not want us to come to harm."
  • (which entity?)

• "Self-ceneteredness and self-righteousness may be traits persent in all humans to some degree"
  • (misspelled "present")

• "AI researcher Eliezer Yudkowsky, of the Singularity Institute for Artificial Intelligence, has formulated a good attempt at a set of guidelines for raising the likely of benevolence in superintelligence. He calls these "Friendliness", described at length in an online book-length"
  • (remember that Eliezer is very aware of the flaws in these guidelines)

• "so that every being from unaugmented humans to the largest superintelligences can coexist peacefully and engage in unconstrained exploration and fulfilling experiences."
  • (there are some constraints that are desirable, like not being able to accidentaly hurt yourself)

• "but in broad-brush terms"
  • (I've never heard that term before.)

• "whether like it or not"
  • (typo)

• "Attempting to slow or abort a Singularity would only increase the risk of a malevolent Singularity coming into being,"
  • (consider replacing the word "malevolent")