I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change.
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.
I've had a completely different experience. I usually know the problematic part and then ask the AI to optimize that exact part with benchmarks, telling it to try various different approaches, and it very often gains massive performance increases on various tasks, with me of course steering it and giving it pointers every now and then.
I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.
I did a lot of SQL optimisation with claude code over the last year, what made it work for me was making sure I verify everything with explain analyze. The whole loop was not fully “agentic” because of this but it was still faster and better than when done entirely manually. Claude also suggested plenty of small optimisations I would otherwise not do.
Which model/agent/harness tool did you use? I've found what you describe was my exact experience some ~6-8 months ago, but since about a month or so the game has completely changed. Using 5.6 Sol with highest reasoning setting in Codex or Fable in Code, the models come up with a list of possible improvements from static analysis (ranked by complexity/benefit), write and run their own custom profilers and deliver significant performance improvements with barely any input needed from my side. So this is no longer a model issue, it's a user toolchain issue.
The mistake there is to point it at code to figure out performance optimizations.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
> The place to find them would be performance profiles, query plans, telemetry.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
> there's really no AI needed once this information is available
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
True, but at least IME, once you get beyond fixing the 'obvious bugs' (how many there are depends on the initial quality of the code base - but I agree that AI is really useful to find those), you'll get into a murky grey area of "maybe false positives" which require a lot of time to validate. As a result you sink a lot of time creating and tweaking code-base-specific rules to try filtering out false positives, and IMHO this is exactly the tipping point where the whole thing becomes pointless because it becomes a bureaucratic monster.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
As others have said, these were uninformed guesses which LLMs seem still to be willing to hand out deliberately. If you provide more information (your schemas, the distribution of values in your database tables, the EXPLAIN ANALYZE outputs, DB configuration etc.) it will do a much better job.
The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.
Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.
I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.
Then you're not grounding it to reality properly. LLMs get bad pretty quickly when you just tell them to pull the answer out of thin air. Pin them to reality with actual performance tests to run to test theories and the results will be much better.
Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.
Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.
I am actually using AI to learn about Windows OS performance. I have a couple of prompts scheduled to research, test, and implement performance gains in the OS, then post them here [0] and here [1]. I always admired Brendan Greg and Bryan Cantrill at Sun/Illumos/ for their work on OS performance but never had the time to sit down and catch up to their level due to my time in the military.
I got Codex to find and apply a bunch of settings I didn't know about and in the future I hope to use AI to make real changes in performance that venders probably don't have time or resources for.
that being said, what I've learned so far is essentially
WhyIsItAlwaysHN's entire comment "The place to find them would be performance profiles, query plans, telemetry."
I just find it so frustrating that for a user interface that was basically solved in the 90s, Windows GUI still struggles to survive on hardware exceptionally more powerful than what we had 30 years ago
Not that I don't believe its possible to fix a lot of bugs, I also wonder what the actual dynamic was. Were the people in team working much more than usual as well? Given its Google, I wouldn't be surprised if there was an "internal push" to fix more bugs over next X sprints so that they can publish this blog and some manager can show impact and AI adaption to his superior.
I would guess they have been confronted with an initial flood of newly discovered bugs whenever they added a new analysis tool or approach (e.g. automated static analysis, fuzzing, ...). Once working through that initial flurry of newly discovered bugs one would assume that the frequency goes down again.
Google's entire modus operandi has been "automate everything" for decades. They've been doing this with fuzzers, with project zero and so on. Adding LLMs on top is a very obvious next step. And LLMs improving and finding more bugs also follows. Then improve the harness and the dev tools, to better use the LLMs. And then everything together end-to-end to find-triage-fix-confirm. Your LLMs are as good as the loop they run in, and the loop is as good as the verifier. Seems a reasonable enough dynamic without (or despite, depends how cynical you wanna be) the need for managers to show number goes up on some chart.
1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker.
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
AI critique often funnels itself into a narrow bucket: creating code blindly with AI is bad. That's easy to grant.
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
Yes. AI is a tool, it’s supposed to be used a certain way, anything else is a misunderstanding og what AI is. You have to aim it in the direction you want it to go, not expect it to solve all your problems magically
A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing.
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
> A lot of people here seem to be living in a different universe than me
I feel this way on this topic too.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.
My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.
That's something those products need to figure out, not the users, one potential solution is to erect paywalls and find out how much those products are actually worth to their users ;)
I've mostly only used Codex for reviews, for Godot/GDScript code, and it helped me catch bugs that would've taken me ages to even notice on my own; games be tricky like that.
Claude was mostly useless or annoying up until the last time I tried it (about 3 months ago)
You have to be vigilant though: Codex often picks out obscure edge cases and suggests adding multiple new functions and flags to avoid issues that would be better off left as fast-failure crashes. Maybe that's because of the 5.6 Sol Max I leave it on.
The biggest bug still has not been fixed here, which is:
Google.
We really need an alternative to this greedy and evil corporations
de-facto controlling a huge portion of the modern www stack. All
decision-making processes are here subjugated to what fits Google's
adEmpire. This is a perpetual system of control amplifying abusive
systems in place. We already see this with the recent age sniffing;
Google just yesterday announced that all android users must hand
over their age to everyone else. Before that google kicked out
or locked out open source alternatives to android (or, at the least,
made their life significantly harder than before). These are not
isolated ways of abuse - this is systematic abuse.
We really need alternatives to Google here. It can not be that one
tyrant company dictates so much of people's open lives (and no,
Firefox is no alternative; Mozilla is basically a nerfed and bribed
entity that has given up on firefox many years ago already; ladybird
may become an alternative at one point in time, but right now is not
and there are questions over how Kling handles the overall project,
but these complaints are significantly below what Google is doing
globally here).
Fixing more bugs in the adChromium code base does not change the underlying problem at hand.
I don't know about Chromium specifically, but in general Google requires 2 humans to have looked at every change (normally one is the author, but not necessarily for bot-authored changes).
Seems like they have enough AI they could find a way to put ManifestV2 back into the browser and fix things we actually wanted instead of putting more bullshit "AI improvements" we didn't?
A company valued at $4+ trillion based on the recent AI hype alone is touting the benefits of using said AI, and we're supposed to take this shit for granted.
I'm wondering what the Alphabet employees still commenting on this forum have to say about it? I guess for the right comps they can keep their mouths shut no matter the high level of idiocy involved.
I’ve heard from people inside Google that they’re prohibited from using anything but their own models. AlbertaTech on YT, ex-Google working on YT said in a video that right before she quit someone had threatened to quit if they couldn’t get access to Claude Code
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.
Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.
I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.
Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.
I got Codex to find and apply a bunch of settings I didn't know about and in the future I hope to use AI to make real changes in performance that venders probably don't have time or resources for.
that being said, what I've learned so far is essentially WhyIsItAlwaysHN's entire comment "The place to find them would be performance profiles, query plans, telemetry."
I just find it so frustrating that for a user interface that was basically solved in the 90s, Windows GUI still struggles to survive on hardware exceptionally more powerful than what we had 30 years ago
0 - https://www.lacksan.com/updates/ 1 - https://github.com/Lacksan-Dev/HP-ZBook-Performance
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
That's AI for you.
At Amazon we have many forums to share our AI wins, but none to share AI failures or disappoinments.
No wonder execs make bad decisions regarding AI, they only hear completely one-sided stories.
Because creating 100x more bugs and fixing 100x more isn't something to be proud of.
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
I feel this way on this topic too.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.
My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.
I've mostly only used Codex for reviews, for Godot/GDScript code, and it helped me catch bugs that would've taken me ages to even notice on my own; games be tricky like that.
Claude was mostly useless or annoying up until the last time I tried it (about 3 months ago)
You have to be vigilant though: Codex often picks out obscure edge cases and suggests adding multiple new functions and flags to avoid issues that would be better off left as fast-failure crashes. Maybe that's because of the 5.6 Sol Max I leave it on.
Google.
We really need an alternative to this greedy and evil corporations de-facto controlling a huge portion of the modern www stack. All decision-making processes are here subjugated to what fits Google's adEmpire. This is a perpetual system of control amplifying abusive systems in place. We already see this with the recent age sniffing; Google just yesterday announced that all android users must hand over their age to everyone else. Before that google kicked out or locked out open source alternatives to android (or, at the least, made their life significantly harder than before). These are not isolated ways of abuse - this is systematic abuse.
We really need alternatives to Google here. It can not be that one tyrant company dictates so much of people's open lives (and no, Firefox is no alternative; Mozilla is basically a nerfed and bribed entity that has given up on firefox many years ago already; ladybird may become an alternative at one point in time, but right now is not and there are questions over how Kling handles the overall project, but these complaints are significantly below what Google is doing globally here).
Fixing more bugs in the adChromium code base does not change the underlying problem at hand.
Do you import new more bugs?
I'm wondering what the Alphabet employees still commenting on this forum have to say about it? I guess for the right comps they can keep their mouths shut no matter the high level of idiocy involved.