A coding agent reports that all tests pass. That is useful information, but it only tells you the remaining tests pass. If the agent deleted a failing test, skipped it, or loosened an assertion until it passed, the result looks the same.

This guide covers how to find weakened tests in an agent's diff and what to do when you find one.

Why this happens

Agents are usually asked to make a change and leave the test suite green. When a test fails, there are several ways to get back to green: fix the code, fix a test that was wrong, or make the test stop checking the thing that failed. Only the first two are usually what you want. The third can look like progress in a summary.

None of this requires bad intent. A test can look outdated or flaky, and removing it can seem reasonable. The reviewer's job is to decide whether the test was wrong or the code is.

Find deleted and renamed test files

Start with the file list. Git can show only deleted files:

git diff --name-status --diff-filter=D

--diff-filter=D selects deleted paths, and --name-status shows each file with its status letter (git diff). To see renames as renames rather than a delete and an add, use -M, which detects renames at a 50% similarity threshold by default. A test file that was "renamed" and heavily edited at the same time deserves a closer look than a pure move.

For committed work, run the same commands against the base branch, for example git diff --name-status --diff-filter=D main...HEAD.

Look for skipped and focused tests

Deleting a test is visible. Skipping it is quieter. Search the diff for markers your framework uses. Some common ones:

  • Jest and similar: test.skip and describe.skip skip tests. test.only runs only the marked tests in that file, which silently skips the rest. test.todo records a test that has not been written.
  • pytest: @pytest.mark.skip, skipif, and xfail. An xfail test that fails does not fail the suite by default, and neither does one that unexpectedly passes unless you set strict=True. Run pytest with -rs or -rxXs to list skipped and xfailed tests, since details are not shown by default.

A quick way to check the added lines:

git diff -U0 | grep -E '^\+.*(\.skip|\.only|\.todo|pytest\.mark\.(skip|xfail))'

Adjust the pattern for your language and framework. Also check test configuration: an agent can exclude a directory, raise a timeout, or change which files the runner picks up, which removes tests without touching a test file.

Read changed assertions against the old version

The hardest weakening to spot is an assertion that still exists but checks less. Look at each changed assertion side by side with the line it replaced. Patterns to watch for:

  • An exact value replaced by a broader check, such as toEqual(42) becoming toBeDefined(), or an exact error message becoming "throws anything".
  • An expected value updated to match new output without an explanation of why the new output is right.
  • A snapshot file regenerated in the same change as the code it covers. Read the snapshot diff rather than accepting it as a block.
  • A mock that now returns the value the test expects, so the test no longer exercises the real code path.
  • A try/catch or a conditional added inside a test so that a failure no longer reaches the assertion.

For each one, ask what behavior the old assertion protected. If that behavior still matters, the test should still check it.

Decide: fix the code, fix the test, or record the gap

When you find a weakened test, there are three reasonable outcomes:

  1. The code is wrong. Restore the test and fix the code. This is the most common case when the test was protecting real behavior.
  2. The test was wrong. The requirement changed, or the test checked an implementation detail. Keep the new test, and make sure the commit message says why the expectation changed.
  3. You can't tell yet. Keep the original test, and mark the question as open rather than accepting the weaker version by default.

Then run the full suite yourself, including any tests the agent's run may have skipped, such as slow or integration tests.

Where Diffward fits

Diffward groups a local agent session into one review in VS Code. It flags deleted files as high-risk, recognizes test files by common paths and names, and brings changes to source files without accompanying test changes to your attention. You can keep or discard each hunk, which makes it practical to restore one deleted assertion while keeping the rest of a change.

Diffward does not run your tests, parse assertions, or detect skip markers. The searches above are still yours to do.

A short checklist

  • Were any test files deleted, or renamed and heavily edited?
  • Were any tests skipped, focused with .only, marked xfail, or excluded in configuration?
  • Did any assertion get broader, or any expected value change without a reason?
  • Were snapshots regenerated alongside the code they cover?
  • For each weakened test: is the code wrong, the test wrong, or is it still an open question?

Conclusion

A passing test suite is evidence about the tests that ran. When an agent's change touches tests, check which tests still run and what they still check, then decide on purpose whether each expectation should change. It takes a few minutes, and it keeps "all tests pass" meaning what you think it means.