First – realize what constitutes good tests.

The answer: little (ideally: zero) false positives. Feel free to read my previous article that elaborates on this topic.

But how exactly? Here is my answer:

1. Avoid false positives

That’s tautological, but has to be said.

Anything that can lead to false positives has to be reduced to a minimum. It’s enough justification to any further action. Once you internalize it you might produce the rest of the article on your own.

The counter-example is a beautiful test coverage of 100%, all tests pass, while the app does not work.

Your goal is the oracle that you can trust. Confront your tests with reality and correct them if they produce false positives.

2. Avoid false negatives

Although they aren’t as bad as false positives, you should avoid false negatives too. The oracle shouldn’t be wrong one way or another.

A false negative slows you down but doesn’t directly hurt your customers.

The cutest example of false negatives from my personal career was when the tests didn’t pass… on Mondays. Every other workday they worked fine and made a perfect oracle but Mondays – nah man, it’s all red.

Though you should have a strong preference. False negatives are bad but false positives are your Nemesis.

3. Test user journey

This is the cornerstone argument against class tests.

Users don’t run your classes in isolation.

Users use the app as a subsequent chain of input and output.

Tests that give confidence should mimic this behavior as closely as possible.

Think web app. Users don’t care if a controller works fine. It only matters if a request is processed as expected.

3a. Class != unit

Unit tests are OK but don’t equate “unit” with “class” or “method”.

A unit generally goes beyond one class.

Read this article that explains it more.

3b. End-to-end is king

You should aim for end-to-end tests first and only go lower if you have a good reason (combinatorial explosion, expensiveness).

I don’t hate fast and easy tests like class tests. However, they don’t give enough confidence.

3c. Blackbox setup

Don’t set up the state. Set up the SUT the same as your user would.

Bad: GIVEN user is created in the database...

Better: GIVEN user made a correct register request...

4. Test on production-like env

Notice how this subtly means you should test on something that resembles “environment”. You will need that end-to-end tests, period.

4a. Production-like data

Use as real data as possible on dev envs. Incorrect data used on devs are the source of incidents.

Trick – copy data from production to dev daily. You can anonymize them, if that makes problem.

4b. Ensure specific versions of the packages

In the tests, use the exact packages that your production app will.

Use npm ci instead of npm i and make sure your package.json is not too permissive.

5. Avoid test doubles

Users don’t use doubles in their journey.

Whenever you use a double in your test, you set up a fake stage. This risks a false positive – when your double fulfills the expectations but real dependency does not.

If you can avoid test doubles in your tests – that’s a huge step towards the oracle.

This isn’t to say that you shouldn’t ever use doubles. Rather, don’t overuse them. If you don’t see a good reason why to use one – don’t.

6. Use doubles properly

You will still need some doubles, that’s for sure. The way you write your doubles makes a big difference. 4 subsequent rules are explaining what makes good doubles.

6a. Doubles only on boundaries

There are places for test doubles that makes perfect sense. It’s system boundaries (database, network, file system). The reasons are well-known – predictability and speed.

Using doubles on the boundaries between modules – it’s questionable. A big yes, if modules are cohesive and independent. No, if modules intercommunicate back and forth.

In other words, if you really have modules, use doubles; if your modules are only a facade – don’t bother.

6b. Thin doubles

When you use doubles, make them as thin as possible.

Remember that you’re replacing real code. You better make sure that you’re replacing as little as possible.

In other words, stick to Humble Object pattern.

An antipattern would be a rich repository. When you’re using complex SQL in a layer that will be replaced by doubles – such doubles need to be complex. This again risks false positives.

6c. Contract tests

How do you test doubles?

If you use thin doubles and only in the right places, chances are you don’t need to test doubles.

But, doubles always add risk. If you want to increase your confidence about your doubles, use contract tests.

6d. Prefer fakes over mocks

Mocks test implementation. Fakes patch the gap so that you can test the behavior.

There is this old discussion about London vs Chicago. When it comes to confidence and no-false-positiveness, Chicago wins hands down.

7. Test as little as possible (given enough confidence)

You don’t want to over-test your app. You want some time for writing features. Write only as many tests as it takes to give you confidence.

Don’t cheat! Don’t feel confident if reality shows you otherwise.

7a. Don’t test framework

Start your tests after the framework gives you freedom.

You can assume the framework you use works fine (and other 3rd parties). If it doesn’t and has some bugs – fix them in the framework, rather in your app.

Exception – end-to-end. If you can afford it, treat your framework as one of the layers in your end-to-end tests.

7b. Don’t test data

7c. Don’t test implementation

8. 1 > 0

In the spirit of tule 6, start with one and see what happens. Often a single test is enough as an incidents catcher.

One end-to-end test can catch typos or orchestration incidents.

This is especially useful when you have a test suite full of class tests and never or rarely test the integration. Start with one end-to-end test. It’s often enough.

8a. Boot up the app

This is an especially useful type of “single test”.

Make sure your app boots up – do your docker up on the CI.

8b. New bug = new test

If you want your incident to never happen again, write a test case that covers it.

It’s a super simple yet super useful technique of how to increase confidence over time.

9. Assert important things

You are responsible for recognizing what’s important in your app.

If your logs are important – assert it!

If your data is important and/or often overlooked – assert it! (this is obviously at odds with paragraph 7b but that’s the whole point of this bullet).

9a. Automate what you verify anyway

You’re often testing your app manually. Try to find a way to automate it. If you verify it, it must be important.

Examples:

  • Chances are you test your web app in the browser. Cypress automates user journeys in the browser, that’s a good way to start automating! (remember paragraph 3?)
  • Chances are you’re testing via Swagger – there is nothing simpler than automating it.
  • If you’re constantly console.log intermediate results, maybe write a test and assert their correctness.

10. Think broad

Remember – a false positive is when the app is facing an “incident” but your test had not caught it beforehand.

Think of an “incident” in broad terms:

  • bug
  • missed requirement
  • not enough specification
  • infrastructure failure
  • the app fails to boot up
  • anything that makes your app not work correctly on prod

Try and catch all of them in your test.

10a. Think outside the box

Remember, your goal is to avoid false positives. Your non-goal is to do what others do and write what others write.

Examples:

  • start where you are, add tests progressively (see rule new bug = new test)
  • delete the existing test base if it produces lots of false positives
  • write exploratory tests for that nasty API that changes too frequently

10b. Don’t make excuses

There are many reasons why tests don’t catch incidents:

  • tests are poorly written
  • there aren’t enough of them
  • test environments were different
  • test data was different
  • error in a third-party
  • the incident couldn’t be caught by test.

Only the last one is a fair reason. Don’t overuse it. If something really can’t be tested automatically – always test it manually.

Don’t use other reasons as an excuse. Don’t take defensive approaches “oh, it’s not my job to integrate platforms; mine works as expected“. It’s your job and it’s your fault if something goes wrong.

11. Right-shift your reality check

In security, there is the notion of left-shift security – to catch errors as early in the process as possible.

In terms of confidence, you should think the opposite. The later in the process, the more valuable reality check it makes.

11a. End user first

Your goal is confidence in the app in the market.

Prefer validation from: (the higher the better)

  1. market
  2. end user
  3. stakeholders
  4. PO
  5. QA
  6. development

11b. Early integration

If you develop only one platform (eg. a backend server without control on frontends), aim for as early integration as possible.

There is no way integration goes smoothly and without incidents. Catch them as soon as possible.

11c. Don’t trust tests on dev

Confidence in the early stages (eg. after QA) means nothing if leads to incidents in later stages (eg. on prod).

If you are confident in the lower stages, don’t take for granted that higher ones will go equally well.

12. Know your limits

You can’t tests everything automatically. Aesthetics, UX – these things can’t be enclosed in frames of automated tests.

Minimize the scope of your manual tests but execute them. And execute them honestly.