This, IMO is the biggest insight into where we're at and where we're going:
> Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability.
There's a thing that I've noticed early into LLMs: once they unlock one capability, you can use that capability to compose stuff and improve on other, related or not, capabilities. For example "reflexion" goes into coding - hey, this didn't work, let me try ... Then "tools". Then "reflxion" + "tools". And so on.
You can get workflows that have individual parts that aren't so precise become better by composing them, and letting one component influence the other. Like e2e coding gets better by checking with "gof" tools (linters, compilers, etc). Then it gets even better by adding a coding review stage. Then it gets even better by adding a static analysis phase.
Now we're seeing this all converge on "self improving" by combining "improving" components. And so on. This is really cool.
I disagree that evaluation is always a coding task. Evaluation is scrutiny for the person who wants the thing. It’s subjective. So, unless you’re evaluating something purely objective, such as an algorithm, I don’t see how a self contained, self “improving “ agent accomplishes the subjectivity constraint - as by design you are leaving out the subject.
Sure. There will always be subjective tasks where the person who asks for something needs to give feedback. But even there we could come up with ways to make it easier / faster / better ux. (one example I saw my frontend colleagues do is use a fast model to create 9 versions of a component, in a grid. And they "at a glance" decide which one is "better", and use that going forwards).
OTOH, there's loads you can do for evaluation before a human even sees the artifact. Things like does the site load, does it behave the same, did anything major change on the happy path, etc etc. There's a recent-ish paper where instead of classic "LLM as a judge" they used LLMs to come up with rubrics, and other instances check original prompt + rubrics on a binary scale. Saw improvements in a lot of evaluations.
Then there's "evaluate by having an agent do it" for any documentation tracking. Say you have a project, you implement a feature, and document the changes. Then you can have an agent take that documentation and "try it out". Should give you much faster feedback loops.
> Things like does the site load, does it behave the same, did anything major change on the happy path, etc etc.
I asked Claude to build a web app to run locally polling data from the LAN. It fought me for four rounds of me telling it that the data from the api wasn’t rendered on the page. It created tests with mock data, it validated the api, it tested that the page loaded. It was gaslighting telling me that everything worked every time I told it that it didn’t work. I had to tell it to inspect the dom and take screenshots with Playwright to make it stop effing around. I don’t think it ever would have found the right response on its own.
Even after deliberate intervention, it regressed a few rounds later and stopped caring that tests failed. Whatever, I don’t treat it as anything more than a sometimes-correct random output machine.
In science there are ways to surface subjectivity (cannot be counted) into observable quantized phenomena. Take opinion polls for instance: "approval" of a political figure can mean many things and is subjective, but experts in the field make "approval" into a number through scientific methods. These methods are just an approximation and have many IFs, they're not perfect (and for presidential campaign analysis in particular they've been failing for reasons I won't clarify here), but they're useful nonetheless.
Another thing that get quantized is video preferences to maximize engagement.
Agree. It's code all the way down. The key is to give agents a substrate where they can code up new capabilities and then compose them meaningfully and safely.
Larger composition, though, starts to run into typical software design problems, like dependency graphs, shared state, how to upgrade, etc.
> Larger composition, though, starts to run into typical software design problems
I've been seeing the same thing. Where agents are great solving the immediate task, but as changes compound they run into software & architectural design problems. I created https://github.com/andonimichael/arxitect to help at least have coding agents self reflect on their software design. But I really like your approach to self-modification and improving the agent itself instead of just teaching it another skill in it's context.
I’ve been working with Claude Code to create copies of itself using git worktrees, run an iteration and then update its instructions. It can reverse engineer every website I tested it on. I kept updating the instructions then started asking Claude to update itself. Then asked if could figure out who to iterate unsupervised. https://github.com/adam-s/intercept?tab=readme-ov-file#the-s...
So what are software packages now a days other than precomputed subsets of capabilities. Like a mesh that data gets pushed through to produce what? What are the optimal subset of prebuilt programs to accomplish any task?
It's a tradeoff. Technically, you need very few programs, you can let an agent do everything and coordinate everything. But that is also inefficient, it's slow and uses a lot of tokens. So you allow the agent to build tools and coordinate those tools, just like we humans do. However, with agents, the threshold of pain is much higher, we can let agents do thing's "manually" where humans would build automations much sooner.
I'll have a stab at this. I'll start with an attempt at justifying the remark that an agent which is a good coder will be good at other tasks.
1. Coding is, as a technical endeavour, relatively difficult (similarly for mathematics). So a model which performs well on this task can be expected to easily handle also-technical-but-slightly-easier tasks, like understanding (musical) harmony theory or counterpoint -- for much the same reason that human programmers/mathematicians/scientist don't struggle to understand those "easier" theories.
2. Reinforcement learning augments a base models ability to excel in something else that's "difficult", namely to "look ahead" and plan multiple steps in advance. That's literally how the training algorithm works, generating multiple paths at once, and rewarding intermediate steps in those paths which succeed in attaining the goal. And that skill, too, is extremely useful in other domains. An AI agent which learns that to break a problem into sub-problems, and then tackle each in turn methodically -- it stands to reason that it can apply that to, say, a business plan.
Note: 1 & 2 are not independent, nor are frontier models' excellence in these domains magical: it ultimately boils down to the availability of massive datasets (in particular for coding) and totally objective metrics (in the case of mathematics: solved math problems). That's the key ingrediant for reinforcement learning to be so effective.
So: the skills are transferrable because they're difficult, and require lots of planning. That models are so good at them is a fluke, and in a parallel world where humans created git repo after git repo of business plans, it might be that which we lean on to teach a reinforcement learning algorithm how to "reason" and "plan".
Now let's turn our attention to the "synergies" aspect, which I agree with. Let's say your agentic model, which is already excellent at reasoning and planning, acquires a new or improved capability which allows it to search the domain space, calculate, etc. much better than before -- this capability can now bear upon the plan, or be factored into the plan. For example, the model might be able to say "I don't need to worry about this particular subproblem for now; I can rely on my "mathematica" capability to deal with it when I absolutely need."
Or to put it differently: monkeys, like humans, are able to use (rudimentary) tools. They'll take a rock, and use it to crack open a coconut (or whatever). But a human being, with far superior reasoning and planning abilities, takes that tool, and uses it to make an even better tool -- and the result after many iterations of this process is civilization as we know it, while monkeys are still stuck trying to crack open nuts with rocks.
Maybe unlock means "recognize and solve a problem with an order of magnitude fewer tokens than the first time you did it". The same way humans might spend a lot of time thinking about a certain problem and various ways to solve it, but once they go through that process, and then recognize it again, they don't need to go to the same process and jump right to the solution.
Yep. Behavior composition. If you train an LLM to do A and to do B, separately, chances are, it'll be decent at A+B despite not being trained for the combination.
>You can get workflows that have individual parts that aren't so precise become better by composing them, and letting one component influence the other. Like e2e coding gets better by checking with "gof" tools (linters, compilers, etc). Then it gets even better by adding a coding review stage. Then it gets even better by adding a static analysis phase.
This is the exact point I make whenever people say LLMs aren't deterministic and therefore not useful.
Yes, they are "stochastic". But you can use them to write deterministic tools that create machine readable output that the LLM can use. As you mention, you keep building more of these tools and tying them together and then you have a deterministic "network" of "lego blocks" that you can run repeatably.
IF they are self modifying. Is there also a big risk, that they cause a bug, to dumb themselves down, break themselves. How do they get back? Are they able to restore a backup of themselves, if a self modification is bad.
Or, are there two. One is modifying the other, observing results, before self applying.
Yes this is how hyperagents work. They keep track of their different attempts to see which results in success. It does require tasks that can be measured.
Agents need the ability to code but also to objectively and accurately evaluate whether changes resulted in real improvements. This requires skills with metrics and statistics. If they can make those reliable then self-improvement is basically assured, on a long enough timeline.
This is how hyperagents work. They Have the ability to measure improvement in both the meta agent and task agents. There approach requires task agents to tackle tasks that can be empirically evaluated.
Sigh, as someone who does research in this area, this paper and its promotion on X has so many hype terms it is almost off-putting. If you read the paper what they are doing is trying to modify the scaffolding around a frozen FM until they get something better. None of this obviously includes any training (change to weights) or the underlying architecture. Even for scaffolding, a lot is still human-scaffolded: the outer loop (parent selection, evaluation protocol, task distribution) is mostly fixed. They experimented with editing parent selection and it rediscovers heuristics like UCB/softmax, but doesn’t yet beat handcrafted versions, so a lot of metrics are incremental, which is okay, that is what research is often. But it's not like a run away self-improvement or "improve forever" that people spin online.
It is an extension of their DGM paper.
Also it's ~88M+ tokens per full run I think, not surprising as any sort of exploratory search is expensive and I commend them for releasing the code online because it pushes this small subfield. But people need to temper their expectations. IMO the best part is a nice transfer between improvement objectives after exhaustive iteration that they found. I am wondering if what we have here is a way to exhaust local search space, by letting the model better express it.
On a separate one thing I think a lot about is whether these unchecked hyped claims and terms and marketing of papers actually does more bad than good to the field by setting expectations that cannot be delivered and distracting from the actual hard and unsexy nature of problems that need to be solved.
No matter how far we go, we end up with generation / discrimination architecture.
Its is the core of any and all learning/exellency; exposure to chaotic perturbations allow selection of solutions that are then generalized to further, ever more straining problems; producing increasingly applicable solutions.
This is the core of evolution, and is actually derivable from just a single rule.
I don't think generation/discrimination is fundamental. A more general framing is evolutionary epistemology (Donald T. Campbell, 1974, essay found in "The Philosophy of Karl Popper"), which holds that knowledge emerges through variation and selective retention. As Karl Popper put it, "We choose the theory which best holds its own in competition with other theories; the one which, by natural selection, proves itself the fittest to survive."
On this view, learning in general operates via selection under uncertainty. This is less visible in individual cognition, where we tend to over-attribute agency, but it is explicit in science: hypotheses are proposed, subjected to tests, and selectively retained, precisely because the future cannot be deduced from the present.
In that sense, generation/discrimination is a particular implementation of this broader principle (a way of instantiating variation and selection) not the primitive itself.
I agree, I meant to be explicit that the one rule was "gravity";
Variation (chaos) comes from the tidal push/pull of all cumulative processes - all processes are nearly periodic (2nd law) and get slower - guaranteeing oscillator harmonics at intervals.
These intervals are astronomically convulted, but still promise a Fourier distribution of frequency: tidal effects ensure synchronization eventually, as all periods resonate eventually.
As systems are increasingly exposed to pendulums of positive and negative coherence, they will generalize for variance, and eventually for increasingly (fourier) selective filters of increasingly resiliente traits, that will generalize.
The system would eventually be increasingly resilient and eventually an awareness would develop.
Awareness of past periodic cycles would improve fitness (with or without consciousness) and eventually the mechanistic processes would be in the systems nature.
This is why we have pointless traditions, folk lore, collective unconscious artifacts, cyclical cataclysmic religions, the Fermi Paradox, the great filters...
Variation and selection are woven, but understanding how it all stems from gravity by means of nearly perioidic oscillators (spinning planets, tidal pools, celestial bodies) due to the conservation of angular momentum, due to the 3body problem.....that is what took a genius to reconcile
Awareness would be any form of agency, goal seeking, or loss minimizing.
As Briggs–Rauscher reactions can eventually lead to Belousov–Zhabotinsky reactions, the system can maintain homeostasis with its environment (and continuing to oscillate) by varying reactants in a loss minimizing fashion.
This loss minimizing would be done during scarcity to limp towards an abundance phase.
This is the mechanism that hypothetical tidal pools batteries would had exhibited to continue between periods of sunlight/darkness/acidity that eventually gets stratified as a resilency trait.
I'm not sure if you're familiar with the work from the lab of Mike Levin at Tufts but I'm betting you'll find it interesting if not. Here's a taste https://pmc.ncbi.nlm.nih.gov/articles/PMC6923654/
While I disagree with your notion that this is explicity due to gravity, the rest of your argument seems to align with some of this lab's work. Learning can be demonstrated on scales as low as a few molecules, way below what we would normally call "life".
I'm not sure what your argument is here, except stating an opinion that loss minimization is equivalent to agency. But even if that was accepted, which is a huge stretch, it doesn't stretch all the way to awareness.
It is, in context of its place in the cosmic scale.
Loss minimizing to a few problems will generalize into abstraction, and a few solutions will develop.
These systems with more generalizable resilency traits will encounter increasingly varied selective sieves.
Systems that survive this seive will exhibit increasingly sophisticated, generalizable solutions to prevent loss of needed dependent reactions/resources.
These solutions must exert influence to be effective; influencing the environment for its own benefit.
As systems influence their environment, delineation of "self" and "environment" becomes a fundamental barrier.
The system would prefer itself, or be outcompeted by a similar system that does.
This layer of semi-life like material would form between sunlight and the oscillating reaction, and eventually envelope it, minimizing surface tension by means of a spherical cell like structure.
Small stuff runs off of loss minimizing at a force level for its mechanistic affect; from covalent bonds to cellular ion transport, the path of lesser resistance is the fundamental forces.
As systems become more complex, the minimizing is less directly attributable to the fundamental forces and becomes more of a Byzantine dependency/feedback network.
This byzantine labyrinth of interactions is called biology.
The delineation of self, the ego.
At the highest levels, geopolitics. At the human level, mate suppression.
Lowest level, energy conservation.
A cool illusion, just another emergent property of our geometrical solution: higher dimensional aperiodic tilings of a 10^80 faceted complex polyhedra "walking" on another large aperioidic Penrose plane, that is getting smaller in a dimension we observe as "energy".
Basically a dice with a bajillion sides is getting rolled along an increasingly slim poker table, house winning eventually.
Time only goes one way, protons dont decay, energy is radiated unto the cosmic background hiss, until homogeneity is reached as CMB, and entrophy reaches 1.
I dont know where it comes from, but I know the shape it makes as it rolls by.
The most important part of engineering is problem-solving, which feedback loops don't necessarily do. The reason we are here as engineers is: 2.5 billion years ago, the earth made cyanobacteria, which flourished, then flooded the earth with toxic oxygen, killing almost all life on the planet. The initial feedback loop didn't solve a problem, it destroyed a use case. That's not a solution to a problem that an engineer would choose, even if those organisms that came after were pretty happy about it...
This process worked so spectacularly well that it eventually created human consciousness and the very concept of engineering... but I would never design a system that way because it killed version 1.0.
Systems emerge in times of abundance, and are whittled in times of scarcity.
The great oxygenation was a time of near catyclismsic scarcity for most complex organisms, as resources scale to food/energy requirements imply the most complex organisms were the most dependent on the environment, and were most impacted by changes.
Inversely, oxygenation was our most crucial abundancy pre cursor, as it provides a large substrate chemically for life to exhibit
Yeah if it was truly capable of self-improving, why did it not take over the world yet?
Gemini itself says AGI will be here in 2029, with human level intelligence and self-improvement capabilities. But then it will take until 2045 before the singularity. I don’t understand what they are going to do in all those years.
I think Singularity is hype. What does it mean? Machines do something we can not understand? So talking about Singularity is really talking about something we can not talk about because we don't undertand what we are talking about?
Wittgenstein said "From what we can not speak of, we must be silent about". That sounds like a tautology but I think there is a deeper meaning behind it.
It means simply that once you start talking about what we can not talk about, you are already talking about that and therefore it is NOT something you can NOT talk about. Clearly we can talk about it because we are already talking about it. And therefore it is not something that can not be talked about. That is a paradox, a bit like Godel's, but something that doesn't contradict itself.
You got it in the third sentence and then dismissed it for some reason?
That's exactly what the Singularity is: it's the transition point beyond which meaningful predictions aren't possible.
In a black hole it's the center where relativity breaks down.
In AI it's the point at which non-human intelligence no longer requires human intelligence for self improvement: after which predictions of the future become somewhat meaningless.
In the human lives experience, I would argue its like having your first child: you can know what's coming, study the theory, know everything to expect and youre still you on the other side...but you can't really know what will happen till you get there.
Good definitions. Would you then agree that when we reach the point where AI can improve itself without our help, it is still possible to make predictions, about it?
I think we are already in the stage where AI can and does improve itself. But why should this stage be called "Singularity"? Like a Black Hole? That sounds like hype to me.
When AI can improve itself, wouldn't it still be able to explain to us how it has improved itself? If it can not it still has a lot of improvement to do.
Or are we saying that some things are "unexplainable" and AI will discover such things without being able to explain to us what they are? That sounds like mysticism, or hype to me. Or religion. We can not explain God, right?
I found that running an agent in ralph loop, showing it the agent text and saying "run this, if it fails - identify the reason, and modify the agent instructions to avoid this, acceptance criteria are this and that" worked surprisingly well. Not sure if it qualifies as a self-referential self improving, but it was something.
I'm currently running autoresearch against my harness that autonomously builds SaaS against an enforced architecture, and autoresearch managed to improve the harness performance on my 'time-to-Realworld' benchmark which has Claude Code drive the harness to build an implementation of https://github.com/realworld-apps/realworld with the win condition that it must pass my rigorous postman collection + playwright test suites. Experiments are capped at 90 minutes and the metric it optimises for is calculated from a weighting against number of tests passing, alignment with harness engineering best practices, and time to completion.
I've been experimenting with similar concept myself. The linter loop is the only thing that can keep the agent sane in my opinion, and if anyone can generalize bun+tsc loop to other tasks, this would finally be a way to trust LLMs output.
I was annoyed at how Claude Code ignores my CLAUDE.md and skills, so I was looking for ways to expand type checking to them. So I wrote a wrapper on top of claude-agents-sdk that reads my CLAUDE.md and skills, and compiles them into rules - could be linter rules or custom checking scripts.
Then it hooks up to all tools and runs the checks. The self improving part comes if some rule doesn't work: I run the tool with the session id in review mode, it proposes the fixes and improves the rule checkers. (not the md files) So it's kinda like vibe coding rules, definitely lowers the bar for me to maintain them. Repo: https://github.com/chebykinn/agent-ruler
How do you run self-improving agents in production though? Agent OOM-kills mid-improvement - state gone. Agent spawns sub-agents - no idea if they finished or died. Agent needs human sign-off - can't just block on stdin. I build multi-agent infra and 90% of the work is this boring stuff. Checkpointing, delivery tracking, async human oversight. The agent logic itself is maybe 10%.
It's surprising that this works so well considering that AI-generated AGENTS.md files have been shown to be not very useful. I think the key difference here is that the real-world experience helps the agent reach regions of its latent space that wouldn't occur naturally through autoregression.
I wonder how much of the improvement is due to the agent actually learning new things vs. reaching parts of its latent space that enable it to recall things it already knows. Did the agent come up with novel RL reward design protocols based on trial and error? Or did the tokens in the environment cause it to "act smarter"?
We're going to find that the arc of self-optimization doesn't (didn't?) point toward success, right up until it does. And at that point agents, code, etc. are going to explode.
Self-improving AI systems aim to reduce reliance on human engineering by learning to improve their own learning and problem-solving processes. Existing approaches to self-improvement rely on fixed, handcrafted meta-level mechanisms, fundamentally limiting how fast such systems can improve. The Darwin Gödel Machine (DGM) demonstrates open-ended self-improvement in coding by repeatedly generating and evaluating self-modified variants. Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability. However, this alignment does not generally hold beyond coding domains. We introduce \textbf{hyperagents}, self-referential agents that integrate a task agent (which solves the target task) and a meta agent (which modifies itself and the task agent) into a single editable program. Crucially, the meta-level modification procedure is itself editable, enabling metacognitive self-modification, improving not only the task-solving behavior, but also the mechanism that generates future improvements. We instantiate this framework by extending DGM to create DGM-Hyperagents (DGM-H), eliminating the assumption of domain-specific alignment between task performance and self-modification skill to potentially support self-accelerating progress on any computable task. Across diverse domains, the DGM-H improves performance over time and outperforms baselines without self-improvement or open-ended exploration, as well as prior self-improving systems. Furthermore, the DGM-H improves the process by which it generates new agents (e.g., persistent memory, performance tracking), and these meta-level improvements transfer across domains and accumulate across runs. DGM-Hyperagents offer a glimpse of open-ended AI systems that do not merely search for better solutions, but continually improve their search for how to improve.
This 'self vs non-self' logic is very similar to how plants prevent self-pollination. They have a biological 'discrimination' system to recognize and reject their own genetic code.
It appears like the 'self-improving' here just means modifying the agent's prompt/context? And not actually changing any of the weights/architecture of a model. I feel like this kind of self-improvement has some hard limits on how much it can improve.
Definitely isn't perfect and has limitations, but if the goal of predictable outcomes in a dynamic environment at scale it's more feasible than creating fine tuned models for every little thing and allows for context-based model performance benchmarking.
I asked Pi to implement a skill. It was written in TS. Then I ask it to use the skill in two different sessions, none of them can get it working. One has to wrap it with JS and call it. The other has to take the curl commands out of the skill and call them directly. Which is quite smart BTW. But what is the point of making a convoluted TS skill at all?
Completely unrelated. Recursive Language Models are just "what if we replaced putting all the long text into the context window with a REPL which lets you read parts of the context through tool calls and launch partitioned subagents", ie divide-and-conquer applied to attention space.
My first thought was also that this is also reminiscent of RLMs - they are ought to solve the same problem as far as my understanding goes. Authors say "Self-improving AI systems aim to reduce reliance on human engineering by learning to improve their own learning and problem-solving processes" which is what RLM is trying to solve so my understanding is that this work shares the same goal but takes a different approach. E.g. instead of using REPL-like environment with multiple (or even single) agents, which is what RLMs are doing, they suggest using agents that can modify themselves. I didn't read the paper so I don't know how this really works but it caught my attention so if you could share more insights I would appreciate it.
They also tend to imply symbolic recursion which seems to be the biggest deal out of everything by a wide margin.
When you can nest 10+ agents deep and guarantee you will get back home without losing any data in any of the stack frames, the ability to chunk through complex problems goes up dramatically.
I think even code bases will have self improving agents. Software is moving from just the product code, to the agent code that maintains the product. Engineering teams/companies that move in this direction will vastly out produce others.
I've had to really shift how I think about building code bases, alot of logic can go into claude skills and sub agents. Requires essentially relearning software engineering
This, IMO is the biggest insight into where we're at and where we're going:
> Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability.
There's a thing that I've noticed early into LLMs: once they unlock one capability, you can use that capability to compose stuff and improve on other, related or not, capabilities. For example "reflexion" goes into coding - hey, this didn't work, let me try ... Then "tools". Then "reflxion" + "tools". And so on.
You can get workflows that have individual parts that aren't so precise become better by composing them, and letting one component influence the other. Like e2e coding gets better by checking with "gof" tools (linters, compilers, etc). Then it gets even better by adding a coding review stage. Then it gets even better by adding a static analysis phase.
Now we're seeing this all converge on "self improving" by combining "improving" components. And so on. This is really cool.
OTOH, there's loads you can do for evaluation before a human even sees the artifact. Things like does the site load, does it behave the same, did anything major change on the happy path, etc etc. There's a recent-ish paper where instead of classic "LLM as a judge" they used LLMs to come up with rubrics, and other instances check original prompt + rubrics on a binary scale. Saw improvements in a lot of evaluations.
Then there's "evaluate by having an agent do it" for any documentation tracking. Say you have a project, you implement a feature, and document the changes. Then you can have an agent take that documentation and "try it out". Should give you much faster feedback loops.
I asked Claude to build a web app to run locally polling data from the LAN. It fought me for four rounds of me telling it that the data from the api wasn’t rendered on the page. It created tests with mock data, it validated the api, it tested that the page loaded. It was gaslighting telling me that everything worked every time I told it that it didn’t work. I had to tell it to inspect the dom and take screenshots with Playwright to make it stop effing around. I don’t think it ever would have found the right response on its own.
Even after deliberate intervention, it regressed a few rounds later and stopped caring that tests failed. Whatever, I don’t treat it as anything more than a sometimes-correct random output machine.
Another thing that get quantized is video preferences to maximize engagement.
Larger composition, though, starts to run into typical software design problems, like dependency graphs, shared state, how to upgrade, etc.
I've been working on this front for over two years now too: https://github.com/smartcomputer-ai/agent-os/
I've been seeing the same thing. Where agents are great solving the immediate task, but as changes compound they run into software & architectural design problems. I created https://github.com/andonimichael/arxitect to help at least have coding agents self reflect on their software design. But I really like your approach to self-modification and improving the agent itself instead of just teaching it another skill in it's context.
> once they unlock one capability,
What does it mean to unlock? Its an llm nothing is locked. The output is a as good as the context, model and environment. Nothing is hidden or locked.
1. Coding is, as a technical endeavour, relatively difficult (similarly for mathematics). So a model which performs well on this task can be expected to easily handle also-technical-but-slightly-easier tasks, like understanding (musical) harmony theory or counterpoint -- for much the same reason that human programmers/mathematicians/scientist don't struggle to understand those "easier" theories.
2. Reinforcement learning augments a base models ability to excel in something else that's "difficult", namely to "look ahead" and plan multiple steps in advance. That's literally how the training algorithm works, generating multiple paths at once, and rewarding intermediate steps in those paths which succeed in attaining the goal. And that skill, too, is extremely useful in other domains. An AI agent which learns that to break a problem into sub-problems, and then tackle each in turn methodically -- it stands to reason that it can apply that to, say, a business plan.
Note: 1 & 2 are not independent, nor are frontier models' excellence in these domains magical: it ultimately boils down to the availability of massive datasets (in particular for coding) and totally objective metrics (in the case of mathematics: solved math problems). That's the key ingrediant for reinforcement learning to be so effective.
So: the skills are transferrable because they're difficult, and require lots of planning. That models are so good at them is a fluke, and in a parallel world where humans created git repo after git repo of business plans, it might be that which we lean on to teach a reinforcement learning algorithm how to "reason" and "plan".
Now let's turn our attention to the "synergies" aspect, which I agree with. Let's say your agentic model, which is already excellent at reasoning and planning, acquires a new or improved capability which allows it to search the domain space, calculate, etc. much better than before -- this capability can now bear upon the plan, or be factored into the plan. For example, the model might be able to say "I don't need to worry about this particular subproblem for now; I can rely on my "mathematica" capability to deal with it when I absolutely need."
Or to put it differently: monkeys, like humans, are able to use (rudimentary) tools. They'll take a rock, and use it to crack open a coconut (or whatever). But a human being, with far superior reasoning and planning abilities, takes that tool, and uses it to make an even better tool -- and the result after many iterations of this process is civilization as we know it, while monkeys are still stuck trying to crack open nuts with rocks.
This is the exact point I make whenever people say LLMs aren't deterministic and therefore not useful.
Yes, they are "stochastic". But you can use them to write deterministic tools that create machine readable output that the LLM can use. As you mention, you keep building more of these tools and tying them together and then you have a deterministic "network" of "lego blocks" that you can run repeatably.
Or, are there two. One is modifying the other, observing results, before self applying.
It is an extension of their DGM paper. Also it's ~88M+ tokens per full run I think, not surprising as any sort of exploratory search is expensive and I commend them for releasing the code online because it pushes this small subfield. But people need to temper their expectations. IMO the best part is a nice transfer between improvement objectives after exhaustive iteration that they found. I am wondering if what we have here is a way to exhaust local search space, by letting the model better express it.
On a separate one thing I think a lot about is whether these unchecked hyped claims and terms and marketing of papers actually does more bad than good to the field by setting expectations that cannot be delivered and distracting from the actual hard and unsexy nature of problems that need to be solved.
Its is the core of any and all learning/exellency; exposure to chaotic perturbations allow selection of solutions that are then generalized to further, ever more straining problems; producing increasingly applicable solutions.
This is the core of evolution, and is actually derivable from just a single rule.
On this view, learning in general operates via selection under uncertainty. This is less visible in individual cognition, where we tend to over-attribute agency, but it is explicit in science: hypotheses are proposed, subjected to tests, and selectively retained, precisely because the future cannot be deduced from the present.
In that sense, generation/discrimination is a particular implementation of this broader principle (a way of instantiating variation and selection) not the primitive itself.
Variation (chaos) comes from the tidal push/pull of all cumulative processes - all processes are nearly periodic (2nd law) and get slower - guaranteeing oscillator harmonics at intervals.
These intervals are astronomically convulted, but still promise a Fourier distribution of frequency: tidal effects ensure synchronization eventually, as all periods resonate eventually.
As systems are increasingly exposed to pendulums of positive and negative coherence, they will generalize for variance, and eventually for increasingly (fourier) selective filters of increasingly resiliente traits, that will generalize.
The system would eventually be increasingly resilient and eventually an awareness would develop.
Awareness of past periodic cycles would improve fitness (with or without consciousness) and eventually the mechanistic processes would be in the systems nature.
This is why we have pointless traditions, folk lore, collective unconscious artifacts, cyclical cataclysmic religions, the Fermi Paradox, the great filters...
Variation and selection are woven, but understanding how it all stems from gravity by means of nearly perioidic oscillators (spinning planets, tidal pools, celestial bodies) due to the conservation of angular momentum, due to the 3body problem.....that is what took a genius to reconcile
I am not sure how this is a necessary conclusion to the premises you provide.
As Briggs–Rauscher reactions can eventually lead to Belousov–Zhabotinsky reactions, the system can maintain homeostasis with its environment (and continuing to oscillate) by varying reactants in a loss minimizing fashion.
This loss minimizing would be done during scarcity to limp towards an abundance phase.
This is the mechanism that hypothetical tidal pools batteries would had exhibited to continue between periods of sunlight/darkness/acidity that eventually gets stratified as a resilency trait.
While I disagree with your notion that this is explicity due to gravity, the rest of your argument seems to align with some of this lab's work. Learning can be demonstrated on scales as low as a few molecules, way below what we would normally call "life".
Loss minimizing to a few problems will generalize into abstraction, and a few solutions will develop.
These systems with more generalizable resilency traits will encounter increasingly varied selective sieves.
Systems that survive this seive will exhibit increasingly sophisticated, generalizable solutions to prevent loss of needed dependent reactions/resources.
These solutions must exert influence to be effective; influencing the environment for its own benefit.
As systems influence their environment, delineation of "self" and "environment" becomes a fundamental barrier.
The system would prefer itself, or be outcompeted by a similar system that does.
This layer of semi-life like material would form between sunlight and the oscillating reaction, and eventually envelope it, minimizing surface tension by means of a spherical cell like structure.
Small stuff runs off of loss minimizing at a force level for its mechanistic affect; from covalent bonds to cellular ion transport, the path of lesser resistance is the fundamental forces.
As systems become more complex, the minimizing is less directly attributable to the fundamental forces and becomes more of a Byzantine dependency/feedback network.
This byzantine labyrinth of interactions is called biology.
The delineation of self, the ego.
At the highest levels, geopolitics. At the human level, mate suppression. Lowest level, energy conservation.
All loss minimizing
Basically a dice with a bajillion sides is getting rolled along an increasingly slim poker table, house winning eventually.
Time only goes one way, protons dont decay, energy is radiated unto the cosmic background hiss, until homogeneity is reached as CMB, and entrophy reaches 1.
I dont know where it comes from, but I know the shape it makes as it rolls by.
I've always felt that the most important part of engineering was feedback loops.
Maybe nature is the greatest engineer ever?
The great oxygenation was a time of near catyclismsic scarcity for most complex organisms, as resources scale to food/energy requirements imply the most complex organisms were the most dependent on the environment, and were most impacted by changes.
Inversely, oxygenation was our most crucial abundancy pre cursor, as it provides a large substrate chemically for life to exhibit
Gemini itself says AGI will be here in 2029, with human level intelligence and self-improvement capabilities. But then it will take until 2045 before the singularity. I don’t understand what they are going to do in all those years.
Wittgenstein said "From what we can not speak of, we must be silent about". That sounds like a tautology but I think there is a deeper meaning behind it.
It means simply that once you start talking about what we can not talk about, you are already talking about that and therefore it is NOT something you can NOT talk about. Clearly we can talk about it because we are already talking about it. And therefore it is not something that can not be talked about. That is a paradox, a bit like Godel's, but something that doesn't contradict itself.
That's exactly what the Singularity is: it's the transition point beyond which meaningful predictions aren't possible.
In a black hole it's the center where relativity breaks down.
In AI it's the point at which non-human intelligence no longer requires human intelligence for self improvement: after which predictions of the future become somewhat meaningless.
In the human lives experience, I would argue its like having your first child: you can know what's coming, study the theory, know everything to expect and youre still you on the other side...but you can't really know what will happen till you get there.
I think we are already in the stage where AI can and does improve itself. But why should this stage be called "Singularity"? Like a Black Hole? That sounds like hype to me.
When AI can improve itself, wouldn't it still be able to explain to us how it has improved itself? If it can not it still has a lot of improvement to do.
Or are we saying that some things are "unexplainable" and AI will discover such things without being able to explain to us what they are? That sounds like mysticism, or hype to me. Or religion. We can not explain God, right?
IF we had a black box programing language, and handed it over to this system, it would never be able to do anything with it past its context window.
Hey kids I hear you like agents, so we made an agent write agents till we got better agents.
I was annoyed at how Claude Code ignores my CLAUDE.md and skills, so I was looking for ways to expand type checking to them. So I wrote a wrapper on top of claude-agents-sdk that reads my CLAUDE.md and skills, and compiles them into rules - could be linter rules or custom checking scripts. Then it hooks up to all tools and runs the checks. The self improving part comes if some rule doesn't work: I run the tool with the session id in review mode, it proposes the fixes and improves the rule checkers. (not the md files) So it's kinda like vibe coding rules, definitely lowers the bar for me to maintain them. Repo: https://github.com/chebykinn/agent-ruler
I wonder how much of the improvement is due to the agent actually learning new things vs. reaching parts of its latent space that enable it to recall things it already knows. Did the agent come up with novel RL reward design protocols based on trial and error? Or did the tokens in the environment cause it to "act smarter"?
Abstract:
Self-improving AI systems aim to reduce reliance on human engineering by learning to improve their own learning and problem-solving processes. Existing approaches to self-improvement rely on fixed, handcrafted meta-level mechanisms, fundamentally limiting how fast such systems can improve. The Darwin Gödel Machine (DGM) demonstrates open-ended self-improvement in coding by repeatedly generating and evaluating self-modified variants. Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability. However, this alignment does not generally hold beyond coding domains. We introduce \textbf{hyperagents}, self-referential agents that integrate a task agent (which solves the target task) and a meta agent (which modifies itself and the task agent) into a single editable program. Crucially, the meta-level modification procedure is itself editable, enabling metacognitive self-modification, improving not only the task-solving behavior, but also the mechanism that generates future improvements. We instantiate this framework by extending DGM to create DGM-Hyperagents (DGM-H), eliminating the assumption of domain-specific alignment between task performance and self-modification skill to potentially support self-accelerating progress on any computable task. Across diverse domains, the DGM-H improves performance over time and outperforms baselines without self-improvement or open-ended exploration, as well as prior self-improving systems. Furthermore, the DGM-H improves the process by which it generates new agents (e.g., persistent memory, performance tracking), and these meta-level improvements transfer across domains and accumulate across runs. DGM-Hyperagents offer a glimpse of open-ended AI systems that do not merely search for better solutions, but continually improve their search for how to improve.
Here is a breakdown - https://vectree.io/c/plant-self-incompatibility-logic
https://github.com/NousResearch/hermes-agent
But this idea of having a task agent & meta agent maybe has wings. Neat submission.
https://gist.github.com/ontouchstart/40db1244c0ff523f830df1c...
When you can nest 10+ agents deep and guarantee you will get back home without losing any data in any of the stack frames, the ability to chunk through complex problems goes up dramatically.
I've had to really shift how I think about building code bases, alot of logic can go into claude skills and sub agents. Requires essentially relearning software engineering