Learning from the frustration of test case debates

What is a test case?

The reason I ask is that recently I have been following (and commenting) on a question in the LinkedIn group Software Testing & Quality Assurance  – ” hi guys do u think that creating Test Cases is must? According to me, creating Test Cases is just a waste of time rather we should utilize the time in doing testing. what is your opinion? ”

At first glance I thought it would relatively *easy* and pick apart this question and the ensuing replies. However, after reading through the comments, I immediately felt frustrated. Why?

Upon reflection, I noticed a couple of things.

First, it helps to view the comments from the start. I had missed the fact that there were something like 100 comments before I jumped in. Realising this would’ve help save the frustration because Griffin Jones said it from comment one

@Pradeep – I forecast that this conversation will become confusing because:

a. people will be using unstated assumptions about what is a “Test Case”. Some people’s working definition will exclude other people’s working definition of the term. This “shallow agreement” problem will not become obvious until comment # 78.

And Griffin’s prophecy came to pass.

Which led to *the* problem:

Comments were roughly divided between *a test case is a set of inputs with expected results* group that talked of the test case as a tangible artifact. The second group tended towards seeing the test case as an instance of a test idea and generally speaking this second group were the ones that seem to constantly challenge the assertions of the first group.

And then it dawned on me.

The second group appeared to be aligned with the context driven school of testing and as such realise that were *a lot* of dangerous assertions in the comments made by the first group. For example:

Testcases ensures the tester does not over do his testing and makes sure when and at what stage of his testing he could exit and say that the testing is complete.

If we were to look at the above statement a number of questions spring to mind. First of all, how does a test case ensure that a tester does not over do his testing? What does it mean to overdo testing and if testing is *overdone* what is it compared to be deemed overdone?  If the commenter means ignoring the risk of testing something else or finding information outside of the scope of the test case then overdone has potentially risky consequences for him or her (as they have now have jumped outside of the test case box and may find interesting information…tsk, tsk as now they may not meet their execution test case complete target because now they are THINKING about what they are doing as opposed to just doing *something*.). If the tester became engaged then they would be aware of their coverage and risk model and seek after information that may challenge that model. Notice that the engaged tester does not complete a test just because they have ticked off all of the steps to execute;   otherwise, we end up blindly following a script and we’re checking not testing. This highlights an issue of the commenter viewing a test case as a tangible item when in reality it is an abstraction. It is an idea (or collection of ideas) and *passing* a test case does not guarantee that the idea is finished with. Rather a good tester will most likely extend that idea into many ideas.

Of course we could critically pull apart the rest of the comment and show the fallacy in the statement (such as how does finishing your test cases mean that your testing is complete? It could in some circumstances but I suspect that the commenter meant completing testing – full stop). There are a number of comments like this and they all follow the same theme. We write test cases so that we can cover the requirements and we have repeatable tests so that we can teach others and because the v-model aligns with Saturn and Mercury in the house of Leo – so it must be good!

But I digress…

AND this was  frustrating for me. It seemed that no matter now many times (and in different ways), the second group (lets called them Team CDT) highlighted flaws in the first groups arguments (lets called them Team Factory) then another equally inane comment appears and it made me realise that (to paraphrase James Bach)…

If you are frustrated then it means that something is frustrating!

Realising this then made the rest of the journey…well…more fun.! I realised that I could not wilfully change anyone’s mind except my own. I realised that, regardless of what I shared, others are free to disagree. I realised that no matter how many times I pointed out a fallacy in someone’s argument, it’s up to them if they take heed or not.

AND I realised that I could actually benefit from this and not let the emotion of frustration take hold.

How you say?

By looking for like minded individuals and engaging with them knowing that I’m mostly likely to get a meaningful discussion coming back. By practising pulling apart a comment and challenging someone’s assertions. By applying James Bach’s Huh?Really?So? heuristic and what was initially a frustration quickly became a learning experience.

While it’s galling to see many testers fall into Team Factory, I am hearted to see a number of testers critical of the *status quo* and challenge them (as demonstrated by their replies to team Factory comments). It is through challenging that we grow the craft into something that is stronger, assertive and more critical overall.

KWST#2 – Day 2 – some thoughts

Day 2 had  a different (positive) vibe and in part I think it was due to the solid first time experience reports by Katrina Edgar, Mike Ward and Mike Talks. All three gave reports that really highlighted some of the ethical challenges we face as testers in our day-to-day world.

Some of the thoughts as tweeted on day 2 were…

  • Testing based on rituals is NOT testing – [the blind adherence to a test tool or process or syllabus is NOT testing. Testing is a brain engaged activity]
  • Testers are there to report the truth, not the convenient truth – [Testing is about presenting the facts as they stand and not manipulating them to suit an agenda (for an example of the *truth* being misused see http://www.theaustralian.com.au/australian-it/states-health-payroll-change-was-adopted-untested/story-e6frgakx-1225888223958]
  • If challenging, is your reputation strong enough to withstand any ethical fall out?
  • As a tester have to look through the eyes of the people who matter. Those with whom we have a contract.

During discussion, the topic of agency came up and the point of who do we serve as testers (see http://en.wikipedia.org/wiki/Law_of_agency )? This discussion brought a renewed energy into the room with Geoff Horne leading this. The question that was asked was…

Who do we serve?

Geoff stated…

We are engaged by an individual, and by the organisation behind the individual

Which is true to a point and from the perspective of the engagement between tester and *client*. However, I see the question from the perspective of the tester which is…

I serve myself first before any institution, as I’m responsible for my own ethics

In other words, as a tester, I am responsible for the ethics I hold and I carry. I am responsible for making sure that my house is in order first before the needs of the organisation are considered. At this point, the extension of what I consider ethical is extended to who I serve literally (or in Geoff’s case, from the perspective of the engagement) which are the people I work for/with and the organisation at large.  The discussion of agency and other ethics topics can be summed up quite nicely by a tweet by @NZTestSheep (aka Mike Talks)..

The great thing about an event like #KWST2 is how it challenges our models and maps, and we’re still processing it days afterwards

Learnings:

KWST takes a lot of organising and it is the detail that count such as…

  • A good venue (space, lighting etc)
  • Internet connection ( VERY helpful)
  • Appropriate twitter tag
  • The RIGHT people to invite [This year revealed some really good thinkers and it will be exciting to be working with them at future KWST conferences]
  • It can spawn off-shoots (like David Greenlee’s OZWST)
  • Facilitation is king – it takes practice, a firm hand and the ability to know when to let the conversation flow
  • Preparation before hand FROM everyone (and reminding everyone know that they are potential *speakers*)

Thank you for all those that attended KWST (see http://hellotestworld.com/ and http://martialtester.wordpress.com/2012/06/18/kwst2-what-a-ride/ and http://martialtester.wordpress.com/2012/06/19/kwst2-happy-snaps/ )

Thank you James Bach for your time in helping build a credible, professional, thinking community of testers down under and thank you software Education for your support in hosting KWST#2!

***EDIT: Much thanks must also go to Oliver Erlewein, Richard Robinson and Aaron Hodder for their drive and passion in prompting thinking, engaged testing especially here in Wellington, New Zealand. ***