← All field notes

Independent testing doesn't mean testing alone

IT control frameworks ask for independent testing to protect quality. Worded carelessly, the same control lets development opt out of testing entirely and leaves QA carrying everything. How it happens, who should test what, and what a better control looks like.

19 Nov 202610 min read#Compliance#Process#QA

If you work in QA in a regulated company, this may sound familiar. You're responsible for testing everything. You can't read the source code. You can't call the back-end APIs directly. Nobody can tell you whether unit tests exist, let alone what they cover. So you test the only way that's left to you: end to end, through the UI, with a test stack that gets more complex and more fragile every quarter.

Meanwhile developers ship, quality costs keep climbing, and every incident in production lands on your desk.

It's tempting to blame the developers, or management, or yourself. In most cases I've seen, the cause is more boring and much easier to fix: one sentence in the IT control framework.

I watched this happen in a large financial institution. Nobody there did anything wrong on purpose. The control framework did exactly what it said. It just didn't say enough.

What IT controls are actually for

Control frameworks such as COBIT 2019, and the in-house frameworks built on it, don't exist to assign blame. They exist to make sure work is done correctly, by the right people, and that someone can prove it.

To do that they rely on a simple idea: for every activity, some people are Responsible (they do the work) and one person is Accountable (they make sure the work is done, and answer for it). Several roles can be Responsible for the same activity. Only one can be Accountable.

For testing, accountability is usually split by practice. The practices that cover building the solution make the head of development answer for the technical tests. The practices that cover accepting the change put the business in charge of acceptance testing. The Responsible side is meant to be shared everywhere, because no single role has the skills and the access to test a system at every level.

Accountability can't be shared. Responsibility for testing has to be.

How it goes wrong

Most frameworks contain a reasonable requirement: testing must be performed by someone independent from development. Regulators expect it, auditors check it, and it makes sense: everyone has blind spots about their own work. But independence comes in degrees. Developers reviewing and testing each other's work already find many defects, and find them early. An independent tester adds a different perspective on top of that, not instead of it.

The trouble starts with the wording. I've seen this pattern more than once:

Put those three together and the result follows by itself. Testing, as the organisation defines it, belongs to QA. Developers may still write unit tests, but nobody asks for them, nobody sees them and nobody relies on them. When QA asks for help, "it's not our role" is a perfectly valid answer: the process says so.

The irony is that the testing profession's own reference says the opposite. The ISTQB Foundation Level syllabus (section 1.5.3, Independence of Testing) recommends several levels of independence on the same project: developers performing component and component integration testing, a test team performing system testing, and business representatives performing acceptance testing. It even lists the drawbacks of pushing independence too far: testers isolated from development, an adversarial relationship, testers seen as a bottleneck and, in its own words, "Developers may lose a sense of responsibility for quality."

On paper, the head of development may still be accountable for the technical tests. But once the independence rule defines testing as something development doesn't do, and the build practices ask for no evidence, that accountability is hollow. Nobody reports on it and nobody checks it.

So the people who should make sure testing happens at every level see one team with "test" in its name and conclude that testing is covered. Shared responsibility quietly turns into a single Responsible: QA.

The consequences come in a predictable order:

None of this is anybody's fault in particular. It's a control doing what it was written to do.

Who should test what

Independence is the right principle for checking and accepting the work. It was never meant to stop the people who build the system from testing it. A simple way to put the responsibility back where it belongs is to ask, at each test level, who can actually do the job well. Unsurprisingly, the answer is close to the split ISTQB recommends. The main difference is system testing: depending on its technical complexity, developers may need to be involved to make sure the tests are relevant.

LevelCan the business do it?Can QA do it?Who should be ResponsibleEvidence for the quality lead
Unit / componentNoRarely: it needs the code and development skillsDevelopmentUnit test report from the pipeline, read access to the code for periodic review
IntegrationNoSometimes, depending on access to APIs, networks and security layersDevelopment, joined by QA where access allowsIntegration test report per build, list of the interfaces covered
SystemNoYes, for most of itShared between QA and developmentSystem test report traced to requirements, including non-functional results
AcceptanceYes, it defines what to checkYesBusiness users (for UAT) and QA (for independent acceptance verification)Acceptance report per requirement, business sign-off

Accountability follows the practices: the head of development answers for the technical levels, and the product owner or business owner answers for acceptance. System testing sits in between, so make sure your framework says explicitly who is accountable for it.

Notice the pattern in the table. Deciding what to verify at acceptance level needs no technical knowledge at all: it needs knowledge of the business. Automating that verification is another story. And the closer you get to unit level, the more the work depends on the code, the architecture and the access that only developers have.

A team that leaves all of that to QA isn't more independent. It's just less tested.

What a better control looks like

You don't need to remove independence from the framework. You need to say what it applies to, and to ask the build side for proof. In plain terms, a better control says two things:

The second line is the one that changes everything. It doesn't make QA write unit tests. It makes unit tests visible, so that someone responsible for quality can answer the question every auditor eventually asks: was this requirement tested, and how far?

For each requirement, the quality lead should be able to see:

That last item sounds minor. It isn't. In the organisation I mentioned, QA couldn't read the code of the applications it was responsible for testing. Granting read access costs nothing, and it's often the first step towards a real conversation between QA and development.

Tip: let mutation testing review the unit tests

Reading unit tests by hand doesn't scale, and code coverage won't help much: it tells you which lines were executed, not whether a single assertion would notice if they were wrong. A test with no assertions gets 100% coverage.

Mutation testing answers the real question. The tool makes small deliberate changes to the code, such as turning a > into a >=, removing a call, or returning a constant, and runs the unit tests against each changed version. If a test fails, the mutant is "killed". If they all still pass, you've found behaviour that nobody is checking. The share of killed mutants is the mutation score.

Tools exist for most stacks: PIT for Java, Stryker for JavaScript, TypeScript, C# and Scala, mutmut for Python. Run them in the pipeline on the code that changed, and publish the score next to the unit test report.

This changes the conversation. The quality lead no longer has to trust the unit tests or audit them line by line: the pipeline produces the evidence that they actually catch defects, on every change. Quality stops being something QA inspects at the end and becomes an outcome of the development process itself.

Where to start

If this sounds like your organisation, you probably can't rewrite the control framework on your own, and you don't need to. Start with these steps:

  1. Read your framework's testing controls yourself. Look at the exact wording of the independence requirement. Does it say which test levels it covers? Look at the design and build practices next to it. Do they require any test evidence?
  2. Map who is actually Responsible at each level. Use the table above. If QA is the only name in every row, you've found the problem.
  3. Talk to the Accountable people. Make sure the head of development knows they answer for the technical tests, and that the product owner or business owner understands testing responsibility is shared across several practices. Today, part of it isn't being done or can't be seen.
  4. Ask for one piece of evidence, not a reorganisation. A unit test report in the pipeline and read access to the code are small, concrete asks that nobody can reasonably refuse.
  5. Bring it to your risk or compliance team as a gap, not a complaint. A control that leaves a whole test level unevidenced is a finding waiting to happen. Compliance people tend to listen when you put it that way.

Independent testing is a good principle. It protects quality by making sure someone other than the author checks the work. But independence is about who checks, not about who is allowed to test. When a control blurs the two, QA ends up alone, the system ends up less tested, and the framework meant to protect quality ends up undermining it.

If you're facing this right now and want to talk it through, get in touch.