If AI is a verifiable domain, how do you get AIs to construct benchmarks to apply to themselves?
It seems that the only benchmarks that could be automatically constructed are those for tasks that are themselves verifiable (theorem proving, coding, etc).
Non-verifiable tasks require humans to define what exactly a good job is.
The RSI hypothesis is dependent on an assumption that progress on verifiable tasks generalizes to non-verifiable tasks.
I think the next step for developers is to combine existing technologies. For example, in the medical imaging space that are thousands of published algorithms that were implemented once (by the researcher who published the paper), but never used. AI can now read the paper and one-shot an implementation. I do wonder if there are breakthroughs to be had by combining such algorithms.
Almost certainly yes - I’ve tried this for some imaging problems - but what tends to happen is you find that the method is fragile and only works well on a narrow task, or you need some specific data that’s a pain to label. Reproducibility crisis type stuff. Which makes sense when you think about how narrowly scoped many papers are, and the fallibility of the average grad student. Also survivorship bias is strong. Lots of papers are forgotten about because the methods didn’t really work for other things.
But models are really good at being told “here is a paper, check the implementation against the claims in the paper and munge my data to work with it”. On the plus side, many seminal papers have really simple implementations. Like the meat of many CVPR papers is a derivative repo + one function modified. Often they can fix things or backport improvements from more recent work. And they can definitely re-implement things that ought to work, but the authors never published.
It's coming for everyone: there's a startup that introduced robot arms that promise to learn anything 1 shot: if it holds - and there's little wiggle room for one shot - it would be a good indicator that manual labor is also soon gone.
The only question that remains is if 99.9% of the world population will let themselves be fooled by UBI & VR to keep the 'masses content; while the 0.1% hold all the wealth and AI power.
And this is decided now, because it is too late once AI is consolidated under one bi/tri-llionaire.
Is the equivalent of ants trying to outsmart Google that wants to build a datacenter on their land. ASI will unfathomably outsmart the wealthy and intelligent humans like Stockfish 13 does Magnus Carlsen
This is jumping the real world concerns into the realm of science fiction. How about we engage with the premise that before the AGI god comes into being and annoints the Rationalists as it's chosen people, there will be people who will try and use this to enrich themselves and immiserate others.
Not a chance: if there was even a 1% chance of this being true you'd see something entirely different happening. If AI research self-reinforces, as Anthropic publishes (but doesn't actually believe, it can't), then it will win from all the economic activity on the planet. The major consequence of that is that money (as a source of labor, ie. to get people to do stuff for you, to get stuff for you) is now worthless.
Why? It is now easy to get a few motors to do that stuff for you. It is now easy to get whatever manufactured ... except for raw materials, and energy if you see that as separate.
Think about that again: if superintelligence exists the huge change is that money is now worthless and raw materials have all turned more valuable than their weight in pure gold. Steel, just raw ingots, will increase in price BY A FACTOR ONE HUNDRED THOUSAND, minimum. At this point your dog can probably reason towards the investment opportunity, no?
You would buy farms, mines, mining rights, water, energy (and not for DCs, actual energy sources), rivers, raw materials ... that sort of thing. A superintelligence is not going to be exempt from the means of production. Raw materials are mostly monopolized in the current economic system, so this would in fact be absolutely critical to your success. And hoards of raw materials, anything, will be worth more than Gold (gold being pretty useless except for looking pretty and having a somewhat favorable electrical resistance. So if economic activity is 100x-ing every raw material actually used for something will go into extreme shortage)
So no, sorry. Obviously this cannot be how Amodei, Altman, Google, Meta, and others are seeing things: they're going after money, nothing else. So they are either stupid or they know 99%+ sure they do not have superintelligence.
In other words: AI is a bubble, a hype, according to the management of the big AI firms. It's a way to get people to give them money, not an actual path to superintelligence. Either that, or these people are complete idiots.
He's making an argument and you're free to engage with it or not, if you don't read anything by anyone who doesn't meet your credentials quota then save your time and skip it.
I feel like this is why more distributed control or open ownership of AI models is such an important goal for regulators. As AI accelerates its own development, the private companies that are initially maybe only a few months ahead of open-source will widen the gap more and more.
It seems likely we end up with an even more extreme tech oligarchy than we already have when superintelligence arrives. And that's only if we manage to solve alignment and it doesn't do something more horrible.
Dwarkesh has been skeptical about AI claims for a long time. In his interview with Dario earlier this year, he pressed him hard. So this is a fairly recent change of opinion.
What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.
As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.
The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.
>majority of this acceleration today is likely attributable to Anthropic’s extensive use of AI to do R&D
Lots of companies think they are reaping similar rewards from ai coding and research but it has failed to lift bottomlines or create any new products. My last two employers are gungho on ai and ppl have been producing unfathomable amount of ai output. To a causal observer it would look like we have really accelerated but this has had no visible impact on any of the products that employers produce other than make them shittier.
Whats different about product development at athropic that ai is particular suited for not generating slop.
A lot of AI research can be already automated. The experimental part of work can be tonally outsourced to agents (at least for the topics that I am interested in). I had some success with outsourcing theoretical work as well. These days I use quite advanced orchestration. Interestingly, the most difficult part still is writing good stories for research papers. There I can still beat any AI. And when it comes to research we have been already covered by an avalanche of AI slop. There was always slop in AI research but now it is the fully automated slop. A few years ago there was some human-made slop, now it is being factory produced. AI can do good research on its own, but good research is rare in general.
TLDR; someone used some fake graphs and probabilities pulled out of thin air to justify an unwarranted conclusion based on a false analogy to a situation that is far from clear.
Isn't Sam Altman accused of being a habitual liar by a lot of people, including the richest man on the planet, the arguably best (certainly longest-lasting) AI researcher, the most famous AI ethics researcher, and half of his own board?
Strange how this guy can be CEO of anything. All you hear management types ever blubber about is integrity ... Clearly there's a problem here.
It seems that the only benchmarks that could be automatically constructed are those for tasks that are themselves verifiable (theorem proving, coding, etc).
Non-verifiable tasks require humans to define what exactly a good job is.
The RSI hypothesis is dependent on an assumption that progress on verifiable tasks generalizes to non-verifiable tasks.
But models are really good at being told “here is a paper, check the implementation against the claims in the paper and munge my data to work with it”. On the plus side, many seminal papers have really simple implementations. Like the meat of many CVPR papers is a derivative repo + one function modified. Often they can fix things or backport improvements from more recent work. And they can definitely re-implement things that ought to work, but the authors never published.
The only question that remains is if 99.9% of the world population will let themselves be fooled by UBI & VR to keep the 'masses content; while the 0.1% hold all the wealth and AI power.
And this is decided now, because it is too late once AI is consolidated under one bi/tri-llionaire.
It might all end very badly, but we should not be so prone to presume bad outcomes without good explanations.
Is the equivalent of ants trying to outsmart Google that wants to build a datacenter on their land. ASI will unfathomably outsmart the wealthy and intelligent humans like Stockfish 13 does Magnus Carlsen
Why? It is now easy to get a few motors to do that stuff for you. It is now easy to get whatever manufactured ... except for raw materials, and energy if you see that as separate.
Think about that again: if superintelligence exists the huge change is that money is now worthless and raw materials have all turned more valuable than their weight in pure gold. Steel, just raw ingots, will increase in price BY A FACTOR ONE HUNDRED THOUSAND, minimum. At this point your dog can probably reason towards the investment opportunity, no?
You would buy farms, mines, mining rights, water, energy (and not for DCs, actual energy sources), rivers, raw materials ... that sort of thing. A superintelligence is not going to be exempt from the means of production. Raw materials are mostly monopolized in the current economic system, so this would in fact be absolutely critical to your success. And hoards of raw materials, anything, will be worth more than Gold (gold being pretty useless except for looking pretty and having a somewhat favorable electrical resistance. So if economic activity is 100x-ing every raw material actually used for something will go into extreme shortage)
So no, sorry. Obviously this cannot be how Amodei, Altman, Google, Meta, and others are seeing things: they're going after money, nothing else. So they are either stupid or they know 99%+ sure they do not have superintelligence.
In other words: AI is a bubble, a hype, according to the management of the big AI firms. It's a way to get people to give them money, not an actual path to superintelligence. Either that, or these people are complete idiots.
But we've all seen robots fold clothes to do that sort of thinking easily. Oh maybe we need to hangout on ai twitter more.
It seems likely we end up with an even more extreme tech oligarchy than we already have when superintelligence arrives. And that's only if we manage to solve alignment and it doesn't do something more horrible.
What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.
As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.
The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.
https://www.politico.com/news/2026/09/29/trump-signs-executi...
Lots of companies think they are reaping similar rewards from ai coding and research but it has failed to lift bottomlines or create any new products. My last two employers are gungho on ai and ppl have been producing unfathomable amount of ai output. To a causal observer it would look like we have really accelerated but this has had no visible impact on any of the products that employers produce other than make them shittier.
Whats different about product development at athropic that ai is particular suited for not generating slop.
https://sponsorradar.com/channels/dwarkeshpatel
And that does not include Substack bulk subscriptions.
Trump's mistake is using the word to denote AI in general when it already had an established meaning.
Strange how this guy can be CEO of anything. All you hear management types ever blubber about is integrity ... Clearly there's a problem here.