Software QA FYI - SQAFYI

Testing end-to-end? Remember, more could be counter productive

By:

One of the project I am currently working on had a formal end-to-end testing phase. There are many interrelated systems and end-to-end testing is a good exercise to ensure that system works as expected.

However, itís important to remember that executing more tests, specially during end-to-end testing phase could be counter productive. In general, I prefer less because constraints make me focus on right and important things.

So for end-to-end testing, letís assume that product in question is a web-based e-commerce product such as Amazon. A typical end-to-end test for such a product could be on the lines of

User purchase an item
An order is generated
User gets order confirmation
Inventory is adjusted
Dispatch system gets order information
Dispatch is scheduled
Order status is updated
User gets notification about dispatch
Item is dispatched
Order status is updated
Item is delivered
Order status is updated
User gets an email


A simple user journey such as buying an item triggers action in many connected systems. These systems are often independent and end-to-end testing is performed to ensure that they work together as expected.

So it is important to perform end-to-end testing, but itís equally important to ensure that scope of end-to-end testing is not increased unnecessarily. Unnecessary scenarios included in this phase can increase time it would take to execute them. It is important to understand (and educate others if needed) that purpose of end-to-end tests is not regression or functional testing. Interesting cases related to shopping cart (can add 100 items?) or dispatch / scheduling (can deliver on 29íTh Feb?) etc. are important, but they should be handled during functional testing of the components and not during end-to-end testing.

End to end testing phase should be as small as possible and any request to include scenarios in this phase should be questioned. Value of including a specific scenario should be clear and communicated to everyone. I often ask following questions to see if there are any unnecessary scenarios -

What value are we getting by adding / executing this test?
How this test is different from other tests?
Whatís different about this test and does this different matter?
Is this test uncovering something we donít know already?
Has it been tested before?
Whatís the opportunity cost? Is there anything more important to test?

Remember, executing more tests does not add value and specially in the context of end-to-end tests, executing more tests could be counter productive. Think about the time it takes to set-up data, users, coordinating execution with everyone, monitoring all the messages and ensuring that everything works - Itís time consuming and could be draining.

Always think of the opportunity cost - what else you could be doing if you didnít have to execute your current test? Is there something else more important, useful, valuable you could do? We are already pressed for time and have plenty of challenges already.

Not doing something which could be more valuable can make end-to-end testing counter productive.

Agree? Have you been involved in end-to-end testing phase? What has been your experience? How have you handled it?

Full article...


Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

Testing end-to-end? Remember, more could be counter productive