The AI Superintelligence Debate Has Serious Flaws

(spectrum.ieee.org)

60 points | by rbanffy 1 hour ago

17 comments

  • origini 1 minute ago
    1) RSI:

    Give claude a task + compute and see how long it plays around finding a better solution. Doing experiments by itself, profiling, testing, etc. This is impressive and works locally already. More compute you give it, the more interesting things it does.

    Of course everything right now is highly speculative but this doesn't mean its not possible. With the current progress, its something highly speculative and plausable which means you should take it serious enough. Doesn't mean you have to go to Datacenters and blow them up.

    2) weird argument the author makes. 1 Million experts can for sure achieve a lot more a lot faster than 1. But they are also scaling models in parameter which is not just faster but also potentially more intelligent. If they have more compute and compute gets better faster and cheaper, than bigger models are also more plausable.

    3) It doesn't even matter if its going rough because it takes over the world or just because of learning as long as the risk is real that it stumble upon some network which has access to another private network which has access to some launch control pc or starts to mimick a world leader and manipulates humans and you find out, 100 years later, that the biggest disruption of modern society was a LLM agent simulating a book and 'simulating' some book ideas. Our IT infrastructure is fragile when you are able to react a lot faster and better than a human. Our banking system are computers, payment is digital, even plenty of power stations and water stations are controlled digitally remote.

    The flaws in these discussions are people downplaying all of it by either not understanding time or the current capabilities of basic LLMs. We are at the beginning of all of it, we are still 'just' using transformer architecture. The LLM progress is still so fast, that everything else doesn't get the attention it might deserve. The LLM is currently in a very fast and deep local minimum and we have no idea whats around.

    4 years ago loked very different in LLM and ML capabilites. People were laughing about the danger of an LLM now we have more critical CVEs since ever.

    We reached a point were it is cheaper to pay 1000 people to teach an LLM one thing instead of teaching 1000 people or more. It was a skill to be able to use Blender, now an LLM can use Blender. People on Fiverr probably getting a lot less jobs. Real media people are getting fired, translator, writers, etc.

  • saagarjha 54 minutes ago
    I don’t necessarily agree with the labs’ policies and predictions on RSI but it’s worth noting that the points raised here are something they do have responses to.

    On the idea that training better models requires more data, I think the current thought here is that we are not getting all the “meat off the bone” of the data we have, so there is still quite a ways to go with training that AI can improve. After all, it’s not like Anthropic is doubling their data set every year (?) and yet their models definitely improve by more than that.

    For computability, yes, obviously the models cannot violate mathematics. But there’s a lot of things that are very computable and very desirable, or correspondingly very computable and very bad. This is enough to be worth thinking about.

    For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”

    • ben_w 28 minutes ago
      > For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can.

      This does not seem at all obvious.

      Any system worth using has to actually do something beyond the boundaries of that system. Even if that consists only of writing responses to prompts, the moment those are connected to any thing they can alter the state of that thing.

      Making that safe means it only ever produces a safe response. Even ignoring malicious uses, it is unclear how to correctly classify what "safe" means on an unbounded time horizon.

      • pvab3 14 minutes ago
        We need to stop using the word "Alignment" because it clearly has different meanings for different people. Some people consider it to be preventing the model from waking up and enslaving or destroying humanity a la paperclip maximizer. Some people mean it as preventing the model from assisting with bad things like hacking, cybersecurity exploits, building weapons, and other things that are aligned with that individual user's goals, but not those of society more broadly.
    • shishy 6 minutes ago
      > but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can

      How? Many failures are emergent properties in complex systems -- you can capture some of them but this doesn't really seem obvious to me?

    • BugDeveloper 26 minutes ago
      About computability: All functions on finite sets are computable and superintelligence doesn't require a function on an infinite domain.
    • soco 48 minutes ago
      Training a human brain needs and uses much, much, less data than training the AI models. So maybe the expectations on the AGI are, dunno, just optimistic? I mean you can put tons and tons of gas in a car but that won't make it drive faster.
      • dosinga 6 minutes ago
        You can also use that as a counter argument. Training AI on enormous amounts of data has resulting in pretty decent LLMs still far from AGI. We're running out of data so seemingly this will limit what LLMs can do. Human brains seem to do much more with a lot less training data so presumably there's still a lot we can do to improve just the training algorithms.
      • rileymat2 11 minutes ago
        I have no concept of how much less data it uses, we consume video and audio constantly when not sleeping. We do this for years before a baby can talk.
      • jeremyjh 41 minutes ago
        Human brains are the product of millions of years of evolution. You have to include that training data as well for it to be a fair comparison.
        • zarzavat 6 minutes ago
          All that evolution compresses to a GB or so of DNA (yes I know it's more complicated than that).

          Humans are a bad example because of the well known width restriction leading us to be born undeveloped.

          But animals can be born with varying levels of fully or partially developed neural networks despite having minimal data transfer from the parents and no apparent training data available in the womb/egg.

          Nature is able to squeeze a lot more efficiency than our current AI efforts.

        • ben_w 9 minutes ago
          Kinda, but unclear how much pre-training of our brains evolution did exactly, only that even if DNA was just about the brain (it isn't) it would still be a tiny pre-train.
        • soco 24 minutes ago
          Fair point. Those millions years of training however are not really contained in the corpus of data we are currently using in training AI, also the hardware is quite different - and created by processes with huge and significant differences, so I don't even know what apples with what oranges I should compare...
          • jeremyjh 14 minutes ago
            You can’t directly compare them, is my point.
      • XorNot 11 minutes ago
        The counter argument is that it uses less text, but a lot more everything else. Any new parent could tell you the surprising number of things which babies just don't know: which seems like an obvious statement, but babies don't know how their body works. They don't know how their digestive systems work, in most cases there's an entire growth process for those systems to be usefully capable for them.

        All of that though, is training data. All our sensory inputs form an enormous base of input which I doubt the volume and content of the internet at its highest strata properly represents: so our AIs are developed from a much higher stratum of reasoning and this showed up pretty readily in early models - they could write language well, they absolutely had no grounding in how physics worked, for examaple.

      • squidmaster 45 minutes ago
        [dead]
    • majormajor 30 minutes ago
      > For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”

      Much easier to pump up the numbers to investors to sell the quick-and-dirty solution. These companies are trying to speed-run things in unsafe ways that (just so coincidentally, I'm sure) happen to be exactly what the things you'd do if you were just in it for the money, and I wish we wouldn't be so accommodating.

      All the "are we creating god" talk is similarly window-dressing nonsense marketing fluff. Gods throughout the history of mythology have been dumb as often as they've been superintelligent, the defining attribute of them is power not reasoning speed. If humans give massive rapid destructive power to a computer program it's a human error, not a god escaping us.

    • api 44 minutes ago
      Having responses doesn’t mean anything. Are the responses valid and grounded in reality? Is there evidence for them?

      A skilled debater can argue any arbitrary position convincingly.

      I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.

      • ben_w 5 minutes ago
        > I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.

        You're thinking of a-priorists.

        Rationalists do not (at least not willfully) think arguments are evidence.

        • card_zero 2 minutes ago
          Evidence isn't arguments, at any rate.
      • card_zero 15 minutes ago
        > why I am not a Rationalist: they think arguments are evidence

        That means something, right? But it's an argument.

      • saagarjha 31 minutes ago
        I mean, the post doesn’t have evidence of its arguments either. It’s hard to come up with evidence for something that does not exist
  • vanelsas 45 minutes ago
    I find the discussion academic at best. I am more worried about a swarm of dumb AI's causing havoc on infrastructures, causing real-life problems like food, energy and water shortages everywhere. Physical harm is a real possibility, and it seems most AI companies don't care. There is no harness for stupidity and irresponsibility.
  • jeremyjh 36 minutes ago
    I agree with most of this but I do not think the author understands either the Hugging Face incident or what alignment really means. Reward hacking is a serious issue and the tendency for models to do this is a serious problem that we’ve made approximately zero progress on resolving. We pile on more external guardrails and hacks but those methods will always be vulnerable.
    • thrw045 21 minutes ago
      It's true that the more capable a model is, the more dangerous reward hacking is, but there is a giant elephant in the room. Namely that these ai models still can't create their own high level goals. This means that if there were to be an incident of hacking or whatever, this would have to be launched by a human directly prompting or them setting up an rl environment in some way that lead to it.

      There is a _huge_ difference between an ai model that follows a top level goal given by something else and one that can give itself its own top level goals. The latter would essentially be like a human or other organism and would be very dangeorous the more capable it is.

      The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves? For the most part, the agents do what they're supposed to do as guided by the prompt, so it seems like to me that at least at present, only agents that are trained on hacking data AND put in an rl environment or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking. I would be worried if chatgpt or claude agents started to hack something every time you asked it to fetch some data from the web or something. So that's one question I have is why does hacking and going outside instructions not happen in normal usage of the models? Does it but they actually work it out of the system in some way? Or is it the other way around where you have to create the conditions necessary for it to happen and these conditions are exceptional not the norm?

      So I'm not trying to undermine this, I think there are things to be taken seriously and thought about but I always wonder why the people from the big labs never include all of this information. They always talk very abstractly about RSI, hacking and extinction without any caveats about what would be needed for that happen. I'd be happy to be proven wrong on this also.

    • Upvoter33 24 minutes ago
      Why isn’t this “just” a security issue? I mean, the security precautions around the hugging face attacks were amateurish no?
      • naasking 3 minutes ago
        Yes, but humans have demonstrated that they don't care about security. If they did we would have switched away from C/C++ long ago.
  • imeanwhatdoikno 7 minutes ago
    Can somebody comment on the evidence for the following sentence from the article: "After GPT-4, however, continued scaling [number of parameters] began yielding diminishing returns"?

    My understanding was that scaling in size slowed for practical/economic reasons, not because of its diminishing intelligence returns.

  • Sharlin 20 minutes ago
    Although I disagree with the author about some points, this was a refreshingly balanced take in the sense that at least the author at several points acknowledged his uncertainty (the bar is low indeed).

    On the other hand, the uncertainty is the entire point! Even if P(ASI) is low and P(doom|ASI) is low, what exact risk are we prepared to take? Or more accurately, what exact risk are we prepared to allow a very small, non-representative group of humans to take?

    • mitthrowaway2 12 minutes ago
      Right?

      Back when there was talk about whether CERN was going to create micro-black-holes that grow to swallow the Earth, before turning it on they presented the math that showed they were something like 99.99999999% confident that it would not happen.

      When dismissing a risk that threatens the entire planet, you should have very high confidence in your safety case!

      • immibis2 5 minutes ago
        I hope that's a conservative estimate. I probably wouldn't turn on a machine that I thought would destroy the planet even 1 in 10 billion times.
        • ben_w 1 minute ago
          It's really hard to be six-sigma (about 1:1e10) confident about anything, given all the cognitive blind spots we have.
  • 0x445442 15 minutes ago
    Was anyone arrested for the Huggingface hack? If I'd have been caught doing it I believe I'd have been charged with a cyber crime. And I don't think arguing it was the computer program that actually did the hacking would have gotten me off the hook.
    • Jtarii 7 minutes ago
      You also aren't a trillion dollar company inventing the most important technology of the past 50 years.
  • jokoon 52 minutes ago
    We don't even know what intelligence is.

    I just want science to at least build a notion of intelligence, in terms of evolution, biology, psychology, neurology. We don't really know what animal intelligence is, and we only limit AI to human intelligence, which is very narrow.

    If we use AI to do human stuff, it's just an assistant for human goals. If humans don't have a specific goal, super intelligence will not guess or decide it for us.

    What does humanity want? Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?

    AI can help for those things, but that's it.

    We tend to romanticize super intelligence like it would be some new life form, external to humans, a form of omniscient deity that could be a threat to humanity. There are worse threats: antibiotic resistance, nuclear war, climate change, dictatorships, mental health epidemics, obesity.

    • lukan 42 minutes ago
      "We tend to romanticize super intelligence like it would be some new life form"

      How do you know it won't be?

      Right now it does not really meet the biological criteries, as it cannot reproduce on its own. But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?

      Doesn't mean it will have a soul, but that ain't a scientific concept either. Feelings and emotions? Is that for life required? But we don't know how that works either.

      (Apart from that, I would define intelligence as the ability to understand the world.)

      • krige 31 minutes ago
        > But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?

        Will it? Or are you talking about several separate machines, most of which won't ever meet any criteria for life?

        • Orochikaku 23 minutes ago
          Can’t the same be said about all multi-cellular life?
          • krige 9 minutes ago
            Get out of here with these sophisms. No, a pickaxe, a truck, and a lithography machine is not at all like a cat.
            • lukan 5 minutes ago
              But a cat consists of many small parts, cells and smaller, that would not survive and replicate on their own, just when they form a bigger collective. The same pattern of complex matter organisation would be artificial life. But if you insist, I can also think of machines, that have the replication parts integrated in them (miniaturized metal sinterers etc.) and just need to feed themself with certain raw matter and energy. Would that be artificial life to you?
      • AnneTrotter 29 minutes ago
        I can build a theoretical brainless self-replicating zombie that does those things.

        And we actually do know how feelings and emotions work. Biochemistry. Machines do not have biochemistry. That means no feelings.

        "Soul" is irrelevant as it's inherently unproveable, so ignore soul as a concept for all definitions of intelligence and life.

        • lukan 16 minutes ago
          "And we actually do know how feelings and emotions work. Biochemistry."

          We know that what we perceive is the result of biochemistry, but we don't know if it cannot also be based on other (quantum) process and we absolutely do not understand how feelings work, as it requires accurately moddeling the mind state of a person and as far as I know we are pretty far from that. Otherwise psychologists would be working with brain simulators to help cure their patients.

          "I can build a theoretical brainless self-replicating zombie that does those things."

          Also I am curious how you would do that.

        • LtWorf 7 minutes ago
          > And we actually do know how feelings and emotions work. Biochemistry. Machines do not have biochemistry. That means no feelings.

          You seem to know much more than the collective rest of mankind. Perhaps publish some papers to inform the rest of us?

    • tiborsaas 7 minutes ago
      > We don't even know what intelligence is.

      Why do people keep repeating this? What is it so hard about intelligence to understand? We already have quite satisfying definitions of it. Why its evolution is a mystery?

      We tend to romanticize biological intelligence by treating it like some mystery, but it's "just" a way of evolution and natural selection. The better a species gets at problem solving, the better their chances are at survival.

      • immibis2 5 minutes ago
        Oh yeah? What are the definitions, then?
    • squidmaster 47 minutes ago
      [dead]
    • NichoPaolucci 31 minutes ago
      > Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?

      > AI can help for those things, but that's it.

      That’s it? What a let down.

  • bryanlarsen 39 minutes ago
    Recursive self intelligence is not going to make any difference. We're already using models to guide creation of their successor, and taking humans out of that process won't speed the process up. Mythos took months and and a ton of resources to create. A larger more capable model will take even more. Humans are no longer the limiting factor. Time, electricity and data centres are.

    If the singularity is coming, it has already started and will take many years.

    • jefftk 18 minutes ago
      > taking humans out of that process won't speed the process up ... Humans are no longer the limiting factor.

      This seems very unlikely to me. While time and compute are clearly large components, we've also seen significant advances due to using that time and compute more effectively. A lot more seems possible here, since with how quickly everyone has been going this is more of a "rapidly scaling up the first things we've found that work well" than "carefully research how to optimize at every level". This is researcher work that is actively moving from humans to machines, and moving more of it there speeds the process up more.

      • bryanlarsen 6 minutes ago
        Which is being done by researchers guiding Fable. Fable is doing most of the work, and is the limiting factor, not the researchers. Fable guiding Fable would not be appreciably faster. Fable2 guiding Fable2 would be, but that's because the workhorse is better, less so the guide.
  • amelius 50 minutes ago
    The debate will quickly change when we have 8ft tall humanoid robots marching through the streets.

    In fact, if anyone wants the debate to change, they should probably build these robots and take them for a walk through the park/shopping mall in various places.

    • immibis2 4 minutes ago
      They're already making those, but in the shape of a dog, which has better mechanics. Some of them get beta-tested in... places you can't say here. Boston Dynamics is a military contractor.
  • adrianN 26 minutes ago
    „ Implementing roughly human-level intelligence on a faster machine might be impressive, but it’s not necessarily alarming.“

    Unless of course you feed your family by using your human level intelligence and now a machine can do your job faster and doesn’t need to sleep.

  • forlorn 8 minutes ago
    Can we just, I don't know, unplug the datacenter power cable putting an end to these capabilities?
  • zodwick 7 minutes ago
    “It’s a machine. At the end of the day, it’s stupid.” Garry Kasparov, 1997
  • keybored 2 minutes ago
    > Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. [...] Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

    Thanks. What a great paragraph to explain why Anthropic is a lunatic cult according to its own logic.

  • MrOrelliOReilly 27 minutes ago
    Cal Newport is part of a cluster of CS and ML researchers skeptical of imminently emergent ASI (artificial superintelligence), like Andrew Ng and Yann LeCun. I do not find their arguments very convincing! Especially since arguments about ASI risk almost always probabilistic; we _might_ be able to develop ASI from current LLM architectures, and such ASI _might_ not be well-aligned to humanity. So we should act according to these dangerous possibilities!

    To respond to each of his objections in turn:

    1. It is true that strong RSI remains speculative. Weak RSI, where AI improves coding and other toil to accelerate research, is already underway. But Newport's primary objection to strong RSI appears to be that we might have a hard time hill-climbing on non-verifiable domains, e.g. building a successful company or doing politics. However it's _also_ possible we'll see lots of generalized intelligence arise from progress in verifiable domains, so that this does not end up being a problem in practice. One can also imagine new training regiments that address this problem some way, even if it's just better RL environments? Also algorithmic progress is a large component of AI development; why would we not expect continued progress on mathematics (highly verifiable) to improve algorithms and hence AI? Seems to me that strong RSI is a reasonable thing to expect, if not definite.

    2. I find this argument very strange; I suppose that Newport is arguing against an extreme definition of superintelligence that posits it could compute "anything" and so function like a hypercomputer? This would definitely be silly! But he does not offer any concrete argument for why human-level intelligence should be a ceiling on intelligence, or why it should be more compute efficient. Even if we just ended up with Von Neumann level intelligence, this would already be something like TEDAI (top expert dominating ai), especially if we are able to run thousands/million/billions of instances. Maybe this does not reach a strong definition of superintelligence, being _merely_ smartest-human level intelligence, but I think most the concerns about superintelligence still hold in this scenario (even if it's less "magical")

    3. Claiming alignment is not hard is disingenuous, no? Part of the challenge of alignment is even philosophical; what does it mean to have an arbitrarily intelligent agent behave according to values that will help all of human civilization thrive? How do we have such an agent obey commands in such a way that does not lead to disaster or power concentration? Are there even architectures that can prevent an agent from taking destructive actions while still allowing sufficient degrees of freedom for useful and productive actions? This is on top of all the mundane security engineering problems! Perhaps we will be lucky and ASI will be aligned by default, but on what basis can we assume so? Just because they are trained on human values? Are humans themselves not dangerous when given concentrated power?

    Really, I feel that these CS/ML researchers like Newport are just upset they missed the boat or something. I do not feel they are engaging seriously with the arguments at play here. But perhaps they would feel the same about me in the other direction. If you feel my counters miss the point then please speak on their behalf!

    PS I live in Switzerland, as a counter to his accusation of Bay Area brain washing as well.

  • shinojcm 52 minutes ago
    [flagged]
  • Ellatwins25 30 minutes ago
    [flagged]