Not All QA Tests are Needed, Even If the Agent Says So
Recently, I finished a QA cycle on a bug fix for the integration between our product, for the sake of this article I'll refer to it as Software X, and a third-party platform I'll call Integration Y — specifically, how event titles sync across when the title-sync flag is on. A different company agent wrote up the summary of the code changes and test cases. On the surface, that looks like a routine output from an agent and very common for those using agents to review code and generate QA testing that needs to happen to validate the recent code changed before they make it to production. However, my human reasoning caught an error from the agent.
How agents build test cases
When a bug fix or feature enhancement ships, an agent is remarkably good at deriving the test plan. It reads the changed code (the Pull Request/PR), the ticket, and the acceptance criteria, and it reasons about how the feature is supposed to work now that the bug is fixed or the enhancement is live in the QA environment. From that, it enumerates cases: an event with a long colon-filled title should sync with the colons cleaned to dashes; a title like "Sunday 9:30 Worship" should keep its time colon intact; a contact email with a trailing space shouldn't spawn duplicate contacts on repeated syncs.
Most of those tests were real and valuable. The colon cleaning worked — Integration Y stored the cleaned title, 73 characters, truncated at the field's limit, with dashes where the colons used to be. The 9:30 survived. The trailing-space contact didn't get duplicated. One test case, TC-006, even came with a warning from the agent that time colons might get altered; My agent's QA test disproved that, and then updated the pulse status to "Passed". All of this is what agents do best: reading, understanding, iterating, covering ground, auto updating the pulses in Monday.
But agents don't always produce the needed tests. Three of the nineteen test couldn't be run and only my human judgement was able to identify this and why.
The test that couldn't be validated
One case, TC-009, asked me to sync an event to Integration Y with a missing Field 1, Field 2, or Field 3 — three required fields — and verify the "missing required fields" response.
Here's the problem: I know how our product page was designed. Software X will not let you move along in the process if Fields 1, 2, an/or 3 are not filled out. Adding an event in software X won't save without Field 1 and Field 3 being filled out. When you try to save and move onto the next phase, the "dave" request to the server bounces back with a validation errors. Clearing those fields on an existing event's Details page doesn't even stick; the save claims success and the old values are still there on reload showing in the UI on the browser. And Field 2 isn't a field on the event at all — it's inherited from Field 3, so any event with a Field 3 already carries its Field 2.
Since my agent was auto rolling through the test cases, it tested this anyways. It tried. The steps failed, exactly as I expected them to. I told the agent what I knew about the page's design, and it understood immediately: this wasn't a bug, it was a false test — a test case for a state the application deliberately makes unreachable.
In being a false test, it actually proved the function still worked the way it was initially designed. The validation that blocks incomplete events from saving did its job. The guardrail held. A false test isn't a failed test; sometimes it's a passed one wearing a different label. However, only someone who knew how the page was designed could see that. It was important to share this with my agent since my agent doesn't have acces to the source code and even though the agent that wrote the test cases did, it still produced this test case. Agents aren't perfect and better yet, the human is still responsible for the work.
Knowing the third party
Two more cases — TC-018 (sync all dates stops at the first rejected date) and TC-019 (a failed room create isn't reported as synced) — required Integration Y to reject a date or a room when an event synced over from Software X. I suspected a standard Integration Y setup doesn't do that, because I had a feel for how that third-party software was designed. So before wasting a testing cycle on it, I did some research with an agent and confirmed it: per the vendor's help documentation, Integration Y flags double-bookings rather than blocking them. They surface in a dedicated double-booking view and as an event-conflict warning; overlapping room bookings are left Pending, not rejected. A true hard rejection would require a custom SQL trigger that a customer's own admin adds to their database. We even ran the experiment — a weekly two-date event whose second date double-booked rooms from another event — and Integration Y accepted both dates and booked the same rooms twice without a single complaint.
I gave those findings to my QA testing agent so it could understand the situation now and in future testing of this integration. It folded the research into its write-up on the sub-items in our project tracker, including the reasoning for why these cases were marked "not needed as manual tests" and a note that coverage for the server-side behavior belongs in code-level tests that fake a rejection from Integration Y.
My co-worker appreciated that, because he could feed it to his own agent as context, preventing his agent from generating the same dead-end test cases later. One person's discernment propagated through two agents and a teammate, and it will keep paying off every time either of us tests this integration again.
Knowledge compounds — so does Error
Agents today are fast at reading, understanding, researching, and iterating on their own — and with sub-agents and teams of agents working together, that speed compounds. Agent swarms can churn through a codebase and produce a QA plan in minutes.
Speed was never the question. Judgment is.
Humans are still responsible for what agents produce. We have to be discerning enough to know when a QA test is no good, when an answer is almost right but not quite, when a report seems to be inflating its own numbers. We cannot blindly trust agents and just set them loose to complete work without a human reviewing what they do.
Here's what would have happened if I hadn't caught those false tests: my agent — and every agent downstream that inherited its assumptions — would have continued to believe that Software X events can sync without required fields, that Integration Y does reject conflicting bookings. Those assumptions would have quietly written themselves into every future test plan for this integration. Knowledge compounds. So does error. A false assumption, left unchecked, becomes the foundation for the next hundred test runs, and agents that could have been learning the true shape of the integration would instead be chasing states that don't exist.
Bottom Line
Agents are great co-workers. They can't fully replace people — and honestly, I don't think they should try to. The best arrangement isn't "set the agent loose and trust the report"; it's a partnership. The agent brings speed, coverage, and tireless iteration. The human brings knowledge of the product's design, the third parties it talks to, and the intent behind both — plus the veto when a test case doesn't describe reality.
Three of my nineteen test cases never ran, and they taught me more than the sixteen that passed. They proved the boundaries of the product, not just its features — the validation that refuses to save incomplete events, the third-party system that flags rather than blocks. And they proved something about the work itself: the agent could enumerate the test cases, but only I could tell which ones were real.
That's the job now. Not doing everything by hand, and not handing everything over — but knowing the difference. Agents will keep getting faster. The human in the room is what keeps them honest.
