By: [email protected]

Published August 11, 2025 at 02:40 AM

Agentic Coding Experiment (Mid 2025)

Watch me put different AI models on trial to justify my continued employment

Introduction

Now wait before you run off, I know no one really wants another esoteric post about our AI overlords’ inevitable takeover. As much as I'd like to let my inner Michael Ginsberg go crazy on this, alas, the computers are here to stay.

Anyway, fancy blog intro aside - I'm absolutely exhausted by the "AI Hype™️".

I quote it because obviously opinions really span the spectrum at this point.

I use AI pretty regularly - I pay for ChatGPT Pro as well as Cursor, and while I definitely find value in the tools, I often question their “raw value” - i.e., are these tools actually saving me time, or are they just fun to watch?

I honestly went back and forth a lot (still do).

Recently, Cursor released what they call Background Agents, which, as the name implies, are Cursor agents designed to be run fully autonomously. You give them a ticket/prompt and come back a few hours later to a full Pull Request (PR)!

The dream come true.

Seemed like a great way to see how close we are to the dream of firing every software developer in the world.

Agentic Coding Experiment (Mid 2025) - image 2

Quick Disclaimer

Before getting into this, I want to add a disclaimer - I write this the evening of 8/7/2025, the same day OpenAI released GPT-5, which from my initial tests seems like a definite improvement, but TBD if it's a big step towards “AGI.” I bring this up because things are moving so fast that in the couple of weeks I procrastinated writing this, it’s already marginally out of date.

Also: I make no claims about the future of development, and I’m aware there are privacy implications here. Yes, you can request that services like this don’t train on your code - but who knows what actually happens once you upload it.


Experiment Time

There's a lot of noise around Gen AI right now.

So I thought I'd test the actual goal here: give the agents some work in the morning, check back at night, and see if my job had just gotten a whole lot easier.

Originally, I was going to try on some different repositories/projects, but ultimately it got too time-consuming and this post got too long, so I focused on one.

Agentic Coding Experiment (Mid 2025) - image 5

Round 1 - Next.js Base Blocks

I first tested my NextJS Base Blocks repo - which, shameless plug, you should give a try if you use NextJS!

I picked this repo for a few reasons:

  1. It's public, so I can share the PRs here.
  2. It uses relatively new libraries and packages.
  3. It's intended to be used as a git submodule - being a 'package' makes the development a bit different.
  4. It's a pretty straightforward codebase that I imagine these models were trained on something similar to - i.e., component libraries.
  5. Hate to admit it, but while it's a project that's pretty important to me, I don't love taking time to work on it. Just not the most exciting.

For this one, I ran each agent with both Opus and Sonnet - the former is Anthropic's most advanced coding model at the moment, and spoiler: it does show, but I was curious how these two compared in a practical sense as well, so each run I fed them the same prompt and information to start.

1. Rename bbDropdown to bbSelect

Starting simple - just renaming a component, something any AI coding agent should be able to handle at this point.

  • Sonnet: PR - shorter, got the job done, but weirdly rewrote changelog history in a way that doesn't adhere to my codebase style/usage.
  • While usable, this one would definitely take more time to get it usable than it’s maybe worth for such an easy task.
  • Opus: PR - way more thorough - renamed types, updated demos, docs, added migration notes. Also unnecessarily reformatted stylesheets for no reason.

Winner: Opus for sure, but both worked and were usable with some changes.

2. Improve SCSS variable defaults

Pretty open-ended, opinionated PR. Was curious how it would handle this one - i.e., would it follow my naming conventions? How thorough would it be? etc.

  • Sonnet: PR - passed CI, made a few improvements, took odd liberties with downstream changes.
  • Got CI passing at least lol.
  • Opus: PR - touched every stylesheet, broke the build in a very opaque way.

Winner: Neither. Too much work to fix and didn't really do what I needed. Sad!

With this one I actually tried a bit more after with different prompts, and I just couldn't get anything that seemed worthwhile to try.

3. Update bbNavbar item to accept children

Trickier - big downstream visual/styling changes. Also a little more open-ended exactly how to implement this.

  • Sonnet: PR - added children and dropdownChildren props but didn't edit the title.
  • Added Cypress tests, which was cool.
  • Opus: PR - more aligned with my intent, kept compatibility for client projects, some solid refactoring, but also some big liberties in terms of styling that did break things.

Winner: Opus. Again.

This one was a bit weirder as I don't think either did a great job again. While Opus' was usable, it was a very breaking change if I didn't check it.

4. Convert demo pages to Next.js App Router

Big, open-ended task. Not necessarily difficult, but might require package updates, lots of code changes, etc.

  • Sonnet: PR - totally missed the point - random changes, lots of anys. Beyond a waste of effort and money. CI breaking too, of course.
  • Opus: PR - closer, but 3,400 new lines, mostly from package.json. Broke CI but was fixable.
  • This was definitely pretty impressive that it actually understood what was going on, but it wasn't exactly well done.

Winner: Opus, I guess, merged after lots of fixing.

This did lead to breaks that I'm still noticing. Thankfully this wasn't really important - the routing/app functionality of this repo is purely for demo purposes and doesn't affect downstream projects.

5. Add visual indicators for required fields

Easy to describe, big ripple effects, potentially subtle UI side effects that should be considered.

  • Sonnet: PR - added a weird “required demo” page, plus some new tsconfig file for some reason?
  • Opus: PR - added form component docs, otherwise very similar to Sonnet.

Winner: Opus again, with maybe minimal changes this time, which is nice.

I had to rebase both of these, which was annoying that it wasn't smart enough to do that or even notice.

6. Fill In Remaining Cypress Tests

This one’s a bit different. I asked it to fill in missing Cypress tests and try to get me closer to 100% coverage.

I waited an extra day to check this PR, and it just broke because I waited too long. It was quite weird, but no matter what I did, it just didn't work.

I tried rerunning each model a few more times, and it just refused to work properly - something about re-running was unsuccessful.

To be fair, this is more a product failure than a model failure, but nonetheless, lots of wasted money on this one.


Overall Takeaways:

  • Opus generally delivered more and often noticeably better work - but at higher cost.
  • If you're planning on actually using this work, I think the cost is negligible compared to your time.
  • Sonnet was faster/cheaper but often missed details or made odd decisions.
  • Merge conflicts? All me - no tooling help there.
  • Prompting and reviewing took a surprising amount of time, sometimes more than just coding it myself.

Agentic Coding Experiment (Mid 2025) - image 4

$$$

Cursor charged me about $20 for this batch - closer to $10 if you think of it as “I ran both models for each task.” Not a huge sum, but my product isn’t much better, and I definitely didn’t save time in retrospect.

Self-hosted? I looked into it (Reddit thread, ObservedObserver/async-code) - but at the moment, nothing local is good enough to run completely unsupervised. You’d still be shipping your code off somewhere via an API.

"Hmmm IDK this feels anecdotal"

You’re right - this is very much anecdotal. That’s kind of the point.

I’m less interested in the benchmark charts and more in: does this actually save an experienced developer time?

For me, so far: not really.

It’s also worth noting: - There are privacy and IP implications here. - The model you pick matters. Opus can brute force more, but you pay for it. - Post-merge, I still found small broken things (dropdown menus, alignment quirks) that required follow-up fixes. - If this were my production app, I’d still want to review every line.


Conclusion

It’s hard to have a succinct conclusion about this.

As you might be able to tell, I’m still skeptical. While most of the work was “mergeable,” I’m not convinced I saved time - and I now understand my own codebase less than if I’d written the changes myself, which feels like a net loss.

Also… it was just less fun than doing it myself.

Still - it’s objectively impressive that we can run something like this at all. Even with cost, errors, and the occasional hallucination, this is a new paradigm. I’m curious to see where it goes.

I also wanna be clear - I do enjoy lots of AI tooling available. I use and pay for Cursor and it definitely helps me save time on particular tasks, but I've had to get a feel for what tasks it does well with and more importantly you have to invest time in getting it to work well with your existing repos.

In shortest I think these tools are useful but just perform best on 'greenfield' style / new projects rather than large, existing codebases.

You’ll hear people compare LLMs to a million monkeys with typewriters. Maybe that’s true. But the monkeys are already getting keys to production in some companies.


Thanks for reading - I'd love to hear any and all thoughts in the comments! Is Gen AI dumb? Was my test flawed? Does my code review suck? How’s Gen AI feeling to you in mid/late 2025?

P.S. My favorite Gen AI feature to date remains the “Write commit message” button. The messages are genuinely bad, but they’re still better than my classic stash, save, or arguhibuia.

Cursor Background Agent Performance - image 1

Comments

No comments available :/

Leave your thoughts! 💭


Tags

Agents

Gen AI

ML/AI

Productivity

Say hi!