A discourse on Test Pyramid

“All models are wrong, but some are useful”

George Box

Imagine this. You are working in local groceries shop. Your job is making deliveries to customers homes. You have a beautiful small pickup that serves you well. The car is compact, has a smooth ride and is cheap in maintenance.

source

But as the business grows the situation has changed, maybe your customer base has grown, and now you have to do also deliveries to the countryside. And there is more of them, and your pickup can’t get all the deliveries at once.

So you decided to upgrade your car: you have raised the suspension, put bigger tires so that it will deal better with the poor roads in the countryside. Then you have extended the trunk it can now hold twice as much cargo!

source

You are happy at effect at your work, but as soon as you take it for a ride, there something is wrong…

The ride is not as smooth as it used to be. Maybe it is due to raised suspension? Or the fact that you have much more weight on the back thanks to the new trunk?

Soon you realise the car is performing much worse in a situation where previously it was working great.  

But it is performing better in the new environment, right? Yes, it is doing slightly better, but in reality, you make maybe one or two deliveries countryside in a month, so it is a net negative transaction. Same for the new trunk. Packing your car to the max is putting to much stress on chassis and engine.  In consequence, you have to visit a mechanic much more frequently than before.

Probably right now: „You think what nonsense! This is not Top Gear! Who in the right mind would do that to their car instead of buying a new one that suits their needs better?!”

I know, right? That is exactly my thought on test pyramid: why people try to improve it instead of creating something new that suits their needs much better?

But to make my point clear some explanations:

First,in bellow text if I am referring to „test” – I am mean „Automation Test”, if I want to mention other forms of testing I will refer to them using a more descriptive term – for example as „Human Testing”.

Same to avoid confusion when I will be writing about original pyramid I will be referring to it as „The Test Pyramid” when I will be referring to it derivatives I won’t be using capitalisation

What is Test Pyramid?

Test Pyramid by its creator Mike Cohn

In short, the test pyramid is a model. Which works as thinking tools.

Basically, its goal is to make you think what type of test automation you have and why.

Why test pyramid exist?

Chances are you have still a problem with making developers testing their stuff. But in most cases, they are finally doing at least unit tests.

Unfortunately, in the early 2000s, when Agile started getting traction, nobody knew how to test in this environment. (Agile Testing book wasn’t released until 2009)

Even when we started automating, we were automating our testers job. Ergo we were automating how we are testing  (or were testing back then) mostly e2e via UI. The Test Pyramid was born in that environment, trying to address this issue.

How does it work?

The Test Pyramid works on the assumption that the „bigger” the test is, the slower and harder to maintain it is. That way, the „end to end” automation is at the top of the pyramid.

Unit tests are at the bottom of the pyramid, they are the foundation of the whole construction. The Pyramid argues that if you don’t have a unit test, there is no point in moving up and writing integration and e2e test.

Basically with every test you want to automate you should answer to one question: Can I push it to lower layer?

Same if you have to try to asset state automation in the project:
Pyramid offers a simple solution to check the number of tests.
If you see the problem. it similary offers a strightforward way to fix it:

Just concentrate on lower layers.

The simplicity of the pyramid is its biggest strength being at the same time the most significant weakness.

It offers simple and easy to use the tool, but The Test Pyramid has one and only one answer.

This is an excellent answer when you have no answer to how to deal with automation at all. But it is „baby’s first answer” and it lacks nuance and complexity.

Test pyramid does not answer to question „How to do all the testing?” It has an only a response to „How to deal with test automation?”, and that answer it gives is far from perfect.

But it is good enough if you are starting your adventure in automation.

Yes, it doesn’t provide you with any information about how to write the tests or what you need to write them. It’s not its goal.

You Either Die A Hero…

Most of them are great tools to think about some problems. For example, I love the „Test Filter” for me it’s the best tool for explaining to non-technical people why The Test Pyramid is useful.

That why in an aspect of being a thinking tool, they are great successors.

But where they fall short is on simplicity aspect. For example, let’s take the famous pyramid with human testing cloud – how do you measure this? If you have one hour of manual testing, is it enough to say you have it?  If your manual testing takes more time than automation is it good? – You probably have an answer to this question, but this pyramid doesn’t have that answer. Yes, The Test Pyramid also doesn’t have a solution for the subject of human testing, but it doesn’t try to be competent in this aspect.

I challenge you to look at the pyramids in the gallery below.  While looking at them, try to answer these two question for each one:

  • Is this pyramid providing you simple way of measuring the problem they are trying to address?
  • Does the pyramid provide a simple solution to fix the problem? (Remember simple != easy)

There is also this gallery with collection of different pyramids.

As you can see there is quite a lot of them.

I have to confess:

I’m cheating.

As you probably have noticed, some of them are not pyramids. Maybe they started as pyramid and then evolved, but right now these  few cases provide you with glance at other options.

Is the pyramid still needed?

Short answer: yes. There are still teams that are new to agile and automation, and for them, this tool is still highly recommended and useful. But I think most of us have outgrown usefulness of this model, and we need to look for something new.  

What could be that „new” one? I don’t know; there is quite a long list of different models, maybe one of them will become as popular as a pyramid if you want to see nice lice  Joe Colantonio has one.

What I know, is a pyramid in current form does her job, and attempt of improving it are not successful And are only adding to the confusion and information notice.

So my challenge to you if you are trying to improve pyramid, try to build something new!

Why new model will be good?

I won’t lie there will be many failures. A lot of bad models. I am not oblivious to Survivor Bias, many of the new models will be useless. but somewhere there something new will be born. Ideas are like science theorem then need to battle it out. And I don’t know about you. But I am excited to find what.

Before I finish

I need to clarify something – none of the presented above derivatives from The Test Automation Pyramid are wrong per se. If they work for at least one team they served their purpose. But that doesn’t mean they are actually improvements on The Test Pyra

That’s all for today

I am planning to do something new for a few months. There will be fewer articles but they will be longer and (I hope better quality) This one is the example. Please tell me if you like it. And what do you think about the pyramid? I have presented my point of view what is yours?

And If you want to discus pyramid and different models in person. I will be at PtaQA in Pozań 23 May.

Huge Thank you for Aleksandra Kornceka and Aretti for reviewing drafts you have no idea how much difference their feedback have done.

Addendnum – I am currently reading Experiences of Test Automation: Case Studies of Software Test Automation and it is possible that testing cloud was part of the first draft. Of the pyramid. It doesn’t change much the offical version you can see on Cohn page is without the cloud.

Dodaj komentarz