Testing end-to-end? Remember, more could be counter productive

By:

One of the project I am currently working on had a formal end-to-end testing phase. There are many interrelated systems and end-to-end testing is a good exercise to ensure that system works as expected.

However, it’s important to remember that executing more tests, specially during end-to-end testing phase could be counter productive. In general, I prefer less because constraints make me focus on right and important things.

So for end-to-end testing, let’s assume that product in question is a web-based e-commerce product such as Amazon. A typical end-to-end test for such a product could be on the lines of

User purchase an item
An order is generated
User gets order confirmation
Inventory is adjusted
Dispatch system gets order information
Dispatch is scheduled
Order status is updated
User gets notification about dispatch
Item is dispatched
Order status is updated
Item is delivered
Order status is updated
User gets an email


A simple user journey such as buying an item triggers action in many connected systems. These systems are often independent and end-to-end testing is performed to ensure that they work together as expected.

So it is important to perform end-to-end testing, but it’s equally important to ensure that scope of end-to-end testing is not increased unnecessarily. Unnecessary scenarios included in this phase can increase time it would take to execute them. It is important to understand (and educate others if needed) that purpose of end-to-end tests is not regression or functional testing. Interesting cases related to shopping cart (can add 100 items?) or dispatch / scheduling (can deliver on 29’Th Feb?) etc. are important, but they should be handled during functional testing of the components and not during end-to-end testing.

End to end testing phase should be as small as possible and any request to include scenarios in this phase should be questioned. Value of including a specific scenario should be clear and communicated to everyone. I often ask following questions to see if there are any unnecessary scenarios -

What value are we getting by adding / executing this test?
How this test is different from other tests?
What’s different about this test and does this different matter?
Is this test uncovering something we don’t know already?
Has it been tested before?
What’s the opportunity cost? Is there anything more important to test?

Remember, executing more tests does not add value and specially in the context of end-to-end tests, executing more tests could be counter productive. Think about the time it takes to set-up data, users, coordinating execution with everyone, monitoring all the messages and ensuring that everything works - It’s time consuming and could be draining.

Always think of the opportunity cost - what else you could be doing if you didn’t have to execute your current test? Is there something else more important, useful, valuable you could do? We are already pressed for time and have plenty of challenges already.

Not doing something which could be more valuable can make end-to-end testing counter productive.

Agree? Have you been involved in end-to-end testing phase? What has been your experience? How have you handled it?

Full article...

Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/