AI Code Review Needs More Than the Diff

What makes a good code review?

LLMs write so much code these days that we can’t keep up with it. When a code change is small I can afford to be picky and thorough. When a code change is large I feel like a rubber stamp machine. In these trying times, when developers are able to push out code at breakneck speed with the help of AI, I need to know that my code review tools have some amount of taste distilled into them. Good AI code review is not only a code-analysis problem, but fundamentally a company-intelligence problem and should be learned in the places where real people are doing real work.

What’s in a code review?

Like any engineering problem, we can talk about this in the context of a stack. Today our review stack includes:

• Code: the change itself, plus how the repository works and what depends on it.
• Intent: what the change is supposed to do. Tickets, specs, acceptance criteria, product decisions.
• Policy: how the team actually builds. Coding conventions, architecture, security requirements, explicit review rules.
• Memory: what the company has already learned. Past reviews, incidents, bug reports, decisions that stuck.
• Behavior: what the change actually does. You can't always infer it from the code or the intent, it has to be tested directly.
• Feedback: where us humans act as the guardians of taste. We let the reviewer know what it got right or wrong, and the next review is better for it.

So what makes a review good? It can't bury you in a million nits, and it can't wave through poorly crafted code. The baseline is flagging beyond static analysis: code smells, over-engineering, patterns the team has already decided don't belong here. But there's a hidden part too. Developers absorb docs, discussions, and planning sessions. They spend years reviewing, letting things slip, and paying for it, either fixing the issue or living with the code they approved. All of that compounds into a good reviewer. Usually. You can't pull someone off the street, even a competent programmer, show them a diff, and ask "should we ship it?" They'll look at you and say, "I guess?"

When I'm using agents to work on Ruby code I want their code to look as good as mine. I'm not going to merge their stuff if it's sloppy no more than I would merge the work of a junior developer who has not yet fully internalized our style.

Our starting point: code review tools that don’t know you

We added Greptile back in early January and it was a massive boon to our development quality. We were swamped with code changes and Greptile became our first line of defense. This was the first time I was impressed with an automated code review tool, I had only used GitHub’s basic reviewers like Copilot and CodeQL before and never took the time to tune them. With Greptile we could tune quite a bit but it always felt reactive, where we would make a change that was an over correction just to over correct in the other direction. What we really wanted was a system that learned from us as we talked about code in Slack, when we reviewed each other’s code and left comments, from our Linear issues and bug reports. We wanted something that we didn’t have to tune through configuration and UI.

We started using Adapt for reviewing as well. When we open a pull request in GitHub we have actions that spin up a preview environment. Someone on the team came up with an idea: create a qa user and let Adapt use it to log in and take screenshots of it using the feature. This was massive for us. Jumping from looking at the code to using the code really opened our eyes to how we could create skills to help with automated code reviews. This led to us building coding-style skills and reviewing skills that were built from how we as a team actually wrote software. We have always been plagued with a massive amount of comments, mostly useless, that coding agents would write, and this was a way to combat that.

A few months before writing this, the idea of looping agents took off. Originally with the Ralph Wiggum technique and then evolving into loops. Adapt was the perfect testbed for loops and I created the Adapt Janitor looping task. This looks for places in the code that need to be spruced up: removing comments, adhering to our coding conventions, and uses a separate GitHub repository for its working memory. To date, I have merged every bit of code it has thrown at us. When the auto reviewer blocks pull requests, the janitor uses its next run to fix the PR, full loop.

With the janitor in place, I wanted to build a feature that we would share with all of our customers, an auto-review bot. I took every lesson we learned from the skills we built and turned it into a product that was exposed to our users. Why? If we already had Greptile, why this? Well, in March Greptile changed their pricing. Before early March we paid a flat price for the month to use it, and the price change meant that we could only use 50 reviews per dev per month before being charged $1 per review. This included our janitor and other Adapt-authored requests. We quickly saw our bill skyrocket, and combined with the fact that we were leaning more heavily on Adapt’s reviews, I pushed out the automatic pull request review feature. Our last PR with Greptile was almost 2,000 code changes and just under six months later.

Siloed intelligence

Adapt’s auto review functionality is far from perfect. In fact, next to dedicated code review tools like Greptile it’s a little rough. I have been burning through bug tickets trying to get it to a state where our customers truly love it, but to be honest it isn’t there yet. I think that’s fine because it has something that Greptile doesn’t: It is a part of our business in a way that Greptile couldn’t be. Adapt helps us collaborate, search, and understand all of the data coming into our business. It listens and learns from each employee and is present everywhere we work. I can plan new features with Adapt, have it review docs and specs. It reviews sales calls, reads our external Slack channels where our customers are talking to us. As an engineer I constantly lean on Adapt to understand our customers’ needs and wants and it helps me find the signal in the noise.

This same signal that rises to the top is present in Adapt’s code reviews. When reviewing it has access to our knowledge base that gets populated by just talking to it. Our team has spent time talking in Slack to each other about coding style and then asking Adapt to remember what we said to use later on. We have built skills and hooked up external applications like Linear that it can search through. The same intelligence that powers my janitor loop also powers our code reviews. A good reviewer isn’t siloed from the business, but should be a core part of it. Greptile is an amazing tool, but it wasn’t where we worked, not like Adapt is.

Anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.

Greptile never really made a mistake, but we did need to engineer a solution that allowed us to share what we were doing internally with our customers. We wanted to give everyone a code reviewer that was core to their business, that learned from them. Not a reviewer that had some best practices and some configuration, not something that learned only on GitHub, but something that flexed and bent to their will without imposing ours on them.

This isn’t to say that Greptile can’t use sandboxes, or connect to external apps, or build memory, but it is siloed from the rest of our infrastructure. Adapt gives us a holistic experience where everyone in the company can contribute to the efficacy of Adapt’s reviews. We wanted the reviews to inherit intelligence our team was already shaping.

Bet on LLMs

After more than a decade of required reviews in various codebases, I think there's very little a human can do better in a review than a GPT-5.6. (It's different when reviews are used for sharing knowledge or discussing approaches.)

When it comes to the capabilities of LLMs, we are still in the early years. They will only get better, and at Adapt one of our core principles is betting on exactly that. But notice what Thorsten added at the bottom there: reviews are also how a team shares knowledge and argues about approaches. That's the part a bigger model doesn't give you. Next year's frontier model will read a diff better than this year's, but it still won't know what your team decided in Slack last month.

That's the bet we're actually making. Not that models replace reviewers, but that a reviewer plugged into your company gets better twice: every time the models improve, and every time your team teaches it something. Our reviews didn't get better because we rebuilt a review product feature by feature. They got better because we powered them with intelligence our company was already shaping.

Get a demo of Adapt today and start teaching it your taste.