How Automatic Retests Verify a Fix When Your PR Merges

A fix is not finished when the pull request merges. Somebody still has to establish that the reported problem is actually gone, and that step is often a person re-reading a diff, or it is not performed at all.

Automatic retests close that loop by reading the merged change. Link the pull request that fixes a finding, and when it merges, Odin reviews the code that changed and decides whether the finding is resolved. Here is how the linking works, what the review reads, what it can return, and where it stops.

What the retest actually reads

Start here, because everything else follows from it. Automatic verification reviews the merged code change. Not the running application, and not a replay of the original attempt against a live environment.

That has one important consequence worth stating before the setup steps: automatic retesting depends on a linked pull request, because the retest uses the merged change as its input. If the fix never passed through a pull request, this automatic path is not available.

It also bounds what a passing retest means, which is worth being precise rather than generous about. A verified result says the change addresses the root cause as far as a code review can establish. Automatic retests review the merged code rather than testing the running application.

Three things have to be in place

Connect GitHub. The Borg GitHub App needs to be installed so Odin can read the repository and receive pull request events.

Turn on Automatic retests. It sits on the GitHub card under Management then Integrations, and it is off by default. With it off, merging a linked fix still updates that pull request's state on the finding, but no retest fires. The toggle only appears once retests are available to your organisation, so if the card does not show it at all, that is a thing to ask about rather than a misconfiguration at your end.

Link the pull request that fixes the finding. The fix has to be connected to the finding for any of this to happen, and there are three ways that link gets made.

How a pull request gets linked

Every finding carries a human-readable identifier built from your organisation's prefix and a sequence number, for example BORG-12. It appears in the finding header and on every row of the findings list, and it is the token Odin looks for.

With the GitHub integration connected, Odin can use the finding identifier in the pull request to link the fix to the finding. The identifier can be included in the branch name, title or body.

One detail catches people out. Odin re-scans when a pull request is opened, edited, reopened or closed, and pushing new commits does not re-scan. That is consistent rather than arbitrary: the branch, title and body are the three things it reads, and a new commit changes none of them. So if you forgot the identifier, editing the title or body is what creates the link, not pushing again.

Odin also keeps a single Linked Findings comment on the pull request, listing each linked finding with its severity, status and discovery date. It edits that one comment in place rather than adding a new one on every event.

Why the branch name is the route worth adopting

Of the three, putting the identifier in the branch name is the one that survives contact with a busy week, and the reason generalises beyond this tool.

A link created by the act of doing the work does not depend on anyone remembering to record it afterwards. You copy the branch name off the finding because you need a branch name anyway, and the link then happens on its own when the pull request opens. A link that depends on typing a reference into a description is a link somebody eventually forgets, usually on the busy day when the fix matters most.

That is also why the title and body routes are worth keeping rather than dismissing. They are the repair path. A pull request opened without the identifier is not lost, because editing the title or body re-runs the same detection and links it retroactively.

What linking changes before anything merges

Linked pull requests appear as chips on the finding, one per request, each following its state on GitHub as it moves from open to merged or closed. Hovering one gives the title, repository, author and when it was linked.

The link also moves the finding. A pull request linked to a Reported finding transitions it to Mitigating, which signals that work is under way rather than making any claim about the result. A finding already further along keeps the status it had.

What linking does not do is close anything. Merging a pull request does not by itself move a finding to Fixed & Retested, and that restraint matters in a develop to staging to production workflow, where merging to develop does not mean the fix is live anywhere a user can reach.

What happens when the pull request merges

Given a linked pull request, a finding in a retestable state (Reported, Acknowledged, Mitigating, Open for Retest or Needs Revision), Automatic retests enabled, and the organisation's retest automation available, the merge queues a retest when the finding is eligible. Other conditions can cause a retest to be skipped, such as an already-running retest or insufficient credits.

Odin then checks out the repository at the merge commit, extracts the change the pull request introduced as the diff between its base and head, and reviews that change against the original finding.

Two properties are worth stating plainly for anyone who has to approve this running against their repository. The retest reads a snapshot of the code at the merge commit and then discards it. It never writes to your repository: no commits, no comments, no pull requests.

What the review is actually asking

Four questions rather than one, and the difference between them is where most of the value sits.

Is the root cause addressed. The review reads past the symptom, so a SQL injection finding is checked for parameterised queries rather than for one string having been escaped. A change that makes the reported request fail while leaving the same construction elsewhere is not the same as a fix.

Is the fix complete. It reads the surrounding files rather than the diff in isolation, which is what it takes to see whether a check added on one path is missing on the three other routes into the same code.

Can it be bypassed. A guard placed where it can be reached around is a common shape, and it looks correct in a diff read on its own.

Does it introduce a regression. A fix is a change like any other, and the review treats it as one rather than assuming a security patch cannot break something.

The five things a retest can come back as

Three of them are verdicts. Two mean no verdict was reached at all, and a system that hid those behind a pass or a fail would be misrepresenting how often reading a diff is genuinely ambiguous.

Inconclusive is the row worth reading twice. It leaves the finding open, so you can adjust the fix and merge again, or move it to Open for Retest and have it confirmed by hand instead. Fixed & Retested is the one status only Borg sets, which is what stops the loop closing on an assumption.

Where the result shows up

You do not have to watch for it. In the findings list a chip appears next to the title, reading Retesting while the review runs and then Verified or Retest failed once it finishes. On the finding itself a banner reports the same states, and a finding sitting in Open for Retest with nothing yet reads Not retested yet.

View retest gives the reasoning rather than only the verdict. There is a timeline of how the retest ran, starting with the pull request that triggered it and ending with the code review step and its result. There is also a Spotted while retesting section listing anything else the review noticed in the changed code, each with a file, a line and a short description.

That last part is worth using. A review that has just read the diff and its surrounding files is well placed to notice adjacent problems, and those observations are attached to the retest rather than raised as new findings, so they are easy to miss if nobody opens it.

When it comes back as Needs Revision

An incomplete result is more useful than a failed one, because it arrives with a comment saying what is still wrong rather than a red mark.

The practical loop is short: read the comment, adjust the fix, merge again. A merge of another linked pull request can queue another retest, so the next attempt is verified the same way as the first. Nothing needs re-requesting by hand.

Worth setting expectations with the team on this: Needs Revision is not a rejected fix. It can be a fix that closes the reported path and leaves an adjacent one open, which is the kind of issue the review is looking for and a person re-reading their own diff can easily miss.

Where this stops

Three limits are worth knowing before anyone treats a verified retest as the end of the question.

It reads code, not behaviour. A verified result is a well-informed review of the merged code change, which is a weaker claim than an attempt against the running system that failed.

The automatic retest path needs the fix to arrive as a linked, merged pull request. A change made outside the repository, such as a configuration, infrastructure rule or gateway policy that is managed separately, may leave no relevant diff for that review to read, so fixes of that shape still need verifying another way.

And a merge is not a deploy. The retest tells you the change that merged addresses the finding. Whether that change is running anywhere a user can reach is a question your deployment pipeline answers, not this one.

None of that makes the loop less worth closing. It makes the claim a precise one, which is the difference between a status you can act on and a status you have to go and check.

The other half of this loop, where a finding's status and its ticket in Linear, Jira or GitHub stay in step, works alongside it: the tracker carries the work, and the retest decides when the work is done.

If you want to see this running against your own repository, book a walkthrough with Borg.