Featuring testing luminaries such as Jonthan Kohl, Goranka Bjedov and Karen N. Johnson , STANZ 2011 promises to be THE conference to attend in Australia and New Zealand!

See you there…

The last two months have been exciting for me in the testing space. First it was Rapid Software Testing with James Bach then it was KWST and the follow up online discussions. From these discussions, I have noticed other testers in New Zealand who are thinking about the craft. This is important and I’ll tell you why… In New Zealand there is a problem.

There are not many testers who are:-

• Willing to learn more about the craft
• Promoting better testing practices
• Not promoting ISTQB as the one true way
• Defining what they mean by professionalism (associating professionalism with certification)
• Blindly accepting the status quo

New Zealand is not unique. A lot of testers come from the business and have very little guidance in improving themselves as testers. A fair proportion that I have met are skeptic and critical but a lot are not. They are like sheep willing to adhere to standards or processes or templates as if testing is paint by numbers – produce this 42 page document and hey presto! – instant, credible testing.

Can anyone see the issue with this thinking?

Let’s look the problems I raised above..

There are not many testers who are willing to learn more about the craft.

 Maybe the Department of Statistics can tell us how many software testers have recorded testing as their occupation in New Zealand. I would have guessed a couple of thousand. Out these few thousand, maybe  a hundred or so would be actively involved in improving their testing skill.  Some will be corrupted by managers and HR who know nothing about testing and steer them towards syllabi or outdated practices that offer little value. Some attend conferences (like STANZ) to learn and network and a smaller few will challenge and actively discuss and debate aspects of testing that interest them. The last few are ones that I want to meet because they are the ones that may have an interesting voice and have some influence in their circles they run with (e.g. Oliver Erlewein).

Promoting better testing practices

Looking at testing from a higher perspective, we can do more to promote critical, technical and empirical work. Going to meaningful conferences (exhibition type and invite only peer conferences), participating in online discussions, being coached by James Bach, Michael Bolton or Anne Marie Charrett, attending meaningful training courses, networking – the list is endless.

However, what I’ve seen as common in New Zealand is the blind adherence to standards/non-thinking policy or listening to someone who holds the title consultant (but hasn’t really earned it) who really offers nothing. Here are two tips:-

*If your organisation promotes templates and standards as best testing practice, run.

*If a consultant uses the term best practice, be wary as there is no such thing especially in testing. Be critical of such consultants, challenge them, do not believe them. At best they are marketers not testers.

Promoting ISTQB as the only true way

It seems that management love to quantify and check things off. Certification seems to be something management love. The common responses I hear from management about having certification are:-

• Certification raises professionalism
• And it’s better than nothing

That’s wrong on both counts. Certification does not raise professionalism. Your own reputation and skill as a tester does that (look at the context driven testers like James Bach, Matt Heusser, Pradeep Soundarajan, Henrik Andersson, Anne Marie Charret, Darren McMillan, Richard Robinson , Aaron Hodder, Michael Bolton - (I think I’ve covered the globe!). I can contrast that with testers I’ve interviewed over the last six years that claim to be certified but couldn’t test to save themselves. Unfortunately, these *testers* are a dime a dozen and they are hired as *test* bodies to give the illusion of testing. There is A LOT MORE than certification people! There is skill, experience (good and bad), learning, context and so on.

The argument it’s better than nothing assumes that nothing existed before certification that was of any value. I had been testing a number of years before I even realised that certification existed. Was what I did nothing? Did it have no value? Of course not. One possible answer to this flawed response is to stress reputation and credibility. We may have stumbled in the dark not understanding some academic theory but we weren’t doing nothing. If we have developed a sound reputation then surely something existed before certification that was valuable. If we allow it’s better than nothing argument to foster then we aren’t being critical, thinking testers – testing is more than theory.

Defining what they mean by professionalism (associating professionalism with certification)

This is tied in with the point above. What is professional for me, will be different for someone else. For me, certification does not define professionalism. What defines professionalism is the respect, opinion and reputation of my peers and community i associate with (KWST, context driven school of testing).

Blindly accepting the status quo

Unfortunately, a lot of testers follow what they have been told without question. Most of the time it is organisation based – use this template, report this metric, follow this script. In my experience I don’t see questioning of the status quo to help make the craft better. It appears to me to be ignorance based as in “I don’t know any better.” Ignorance will only take you so far if you wish to be taken seriously as a tester (to me, this type of person is a checker or at best a check analyst, not a tester).

Question the consultants, the test policy makers, the certificationists – there may be legitimate reasons, there may not but at least you’re on the path of refinement.

These are some of the issues that I’ve seen and experienced in New Zealand (and to a lesser extent, Australia). However, there are pockets of testers that are breaking these bonds. KWST is a good example, Sydney Testers meet up and Software Testers New Zealand another. So there is hope in New Zealand and Australia. And where there is hope, there is the opportunity to help influence and shape the testing craft for the better. These pockets are the rebel alliance downunder, we look to counter and debate the threat of the evil empire. You can find them if look below the surface.

We are active in the community, we think, we are critical, we question and align ourselves with communities that hold similar values.

Above all we a passionate about the work that we do!

After the vote, the day went as follows:-

• James introduced information take aways for everyone involved in KWST (for example Cem Kaner’s paper on Recruiting Software Testers and another paper on Exploratory testing dynamics)
• Testing books bibliography (or non-software testing books. Most software testing books are regurgitated folklore).
• An experience report (ER) by James about certification and professionalism
• Followed up by the dice game (a game in problem solving)

Introduction

James introduced some valuable documentation from himself, Cem Kaner, Michael Bolton and others. These documents all tied in with the theme “How i became a test professional or how i helped others become testing professionals”. These documents are a rich resource of information and were gratefully received.

Testing books

We discussed a number of books that relate to testing but are not testing specific (most systematic testing books regurgitate the same thing and don’t offer a lot of value). The books we discussed were books on cognition, human eco-systems, general systems, paradigms and military based.

They had nothing to do with testing and EVERYTHING to do with testing.

An example of one of the books discussed was The book of five rings by Miyamoto Musashi. A literary classic, this book has relevance to testing. There is no right syllabus or right glossary rather it is about finding the most effective way to test – learning many ways, not just one way.

(I will post a list of books we discuss in another post).

ER by James Bach

James then gave a rather interesting perspective on some of the debates/discussions and frustrations he has had with certification programs since the ’90′s (Not just ISTQB). This was an insightful ER in that we could relate to the  effects of overly theoretical, poorly thought out certification program(s) that do not empower testers at all (so James ER resonated a lot with us). In fact, what we are left with are not better skilled testers but indoctrinated subject matter experts. This is a real danger and James ER emphasised this sad fact to many in the room. Not all certified testers value certification but do so because they are motivated by fear on some level. Refer to this great blog post on this topic. James ER fitted in quite nicely with the theme because while we identified the short comings of such certification schemes, we also spent a lot time discussing alternate ways to upskill testers. This was immensely valuable.

It is fair to say that amongst this leadership group is a general uneasiness towards the value of certification – at the very least its nothing more than possum certification, at the worst its giving management and HR the ability to discriminate based on a nothing piece of paper.

This led to a discussion on finding ways that testers can improve their problem finding and problem solving skills – skills that are central to becoming a good tester.

The dice game

James introduced the dice game to the leadership group (I shall not get into specifics here as it may ruin the surprise). This game demonstrates the critical thinking and problem solving that is needed to become a good tester. Four of us already knew the dice game so we acted as observers. Our role as observers was to list the skills and attributes that each team (of 3 or 4) exhibit in trying to solve the puzzle. What was interesting at the end of the game were the range of skills  used to solve the puzzle (teamwork, collaboration, communication, general systems thinking, boundary analysis, equivalence partitioning, data modelling, OFAT, MFAT and so on).

These skills you don’t get on a certification course. These skills are practical, real and meaningful.

Summary

A fantastic two days! A peer conference that was so valuable. It has created a buzz, it has brought together a community, developed leaders, spawn good community building work – it will be something that will happen again!

Thank you first of all to James Bach for his mentoring, guidance and sponsorship before, during and after KWST. Thank you to the 17 software testing leaders in New Zealand who came from all over to attend and thank you to the Software Education team who hosted a magnificent conference.

Inspired by LAWST and under the tutelage of James Bach, 17 leaders met to form a community that, in my opinion, will influence the New Zealand software testing scene for the better. Before i get into KWST itself, it is important to recognise these leaders and influencers.

Aaron Hodder, Andrew Black, Andrew Robins, Bruce Graham, Chris Stapleton, Dion Woodbury, Farid Vaswani, Jeffrey Bedwell, Nadine Brown , Oliver Erlewein, Richard Robinson, Sheryl Toenders, Simon Crutchley, Tad Dawidowski, Tessa Benzie, Brian Osman and of course James Bach.

This is the genesis of a professional, critical testing community here in New Zealand and I am privileged to have an met and learned from every single leader.

This was a two-day peer conference but in this post i will address day one.

Day One – KWST… Started off with an introduction from James about the history of peer conferencing and the method that will be used throughout the two days. After the introduction (and a break), I was first up to deliver an experience report (ER) on:

How I became a test professional or how i helped others become professional

James facilitated this ER (so i could observe how facilitation works) and we used coloured cards to facilitate discussion. During my ER a term came up which we coined possum testing. After a lot of discussion and debate we arrived at three definitions for possum testing. The definition (in the context of my ER) is:

Any testing done that you do not value

in other words, you are like a possum in the headlights driven by fear, stunned, not doing anything of any significant value (e.g. creating 526 test scripts and then following them by rote which what i had on one project – attribution Andrew Robins and Aaron Hodder who I believe initially coined this particluarly definition BUT I attribute to the entire team).

The ER took 20 minutes – the CRITICAL questioning took about two hours! James stressed that at a peer conference, the risk is that your credibility is on the line and you can make or break you reputation. I definitely felt that this was the case. James made another comment which is possibly peculiar to a peer conference in that it’s not the amount you say but it’s what you contribute. I found this to be true in the sense that some of the questions from *quieter* participants were well thought-out and challenging.

We discussed and shared many thoughts on test professionalism which i will share at a later date.

After I was done we took a break and Oliver Erliwein then presented his ER on How to help others achieve test professional by referencing a recent discussion on the google group – software testers New Zealand. This took us the rest of the day as we followed the same format (this time i facilitated and James was involved).

One of the key debates was a discussion on certification (particularly ISTQB as it is the biggest certification scheme in New Zealand) and the lack of skill this certification demonstrates (with numerous examples from the group). This quickly evolved into a rat hole discussion but the general consensus was that current ISTQB certification is, at best, lacking.

HOWEVER, what was more beneficial were the alternatives in which a tester can evolve professionally outside of commercial certification.

These alternatives were:

• Reading books on testing or testing related (e.g. The book of 5 rings or Blink or How to solve it etc)
• BBST series of lecture videos by Prof. Cem Kaner (free and paid for instructor led at a fraction of the cost of current certification schemes)
• Peer conferences
• Networking
• Exhibition conferences (e.g. STANZ) B
• Blogs (creating and reading)
• Problem solving
• Mentoring
• Building your credibility amongst your peers

 These alternatives are not exhaustive however they do highlight one thing – time. You cannot build professionalism in a three-day course – it is built over time.

Andrew Robins made an interesting observation in that he “admires people who do stuff that i admire” which essentially says that a person needs to have time to assess the value of what they are providing. If you value what they are doing then potentially they raise their credibility and hence their reputation.

We also discussed reputation which can thought of as four ideals

• Portfolio – your work
• Reputation – built over time (or lost in an instant)
• Performance on testing – how WELL you do your job *
• Quality of your network – how do you talk and what value do they provide?

(James Bach provided these ideals with Andrew Black and Aaron Hodder providing discussion on the fourth ideal)

This 30 minute ER spawned an incredible discussion (as shown above) well into the late afternoon. We discussed ways in which we can raise the professionalism of our testers that are in our teams or who we influence. By doing so, we fight back against those HR people/ team managers that are looking for the tick in the box. We want testers to be thinkers and leaders not checkers by rote.

As you can see, great discussion, great learning, great sharing!

This was day 1…more on day 2 to follow

KWST 24/25 June 2011:

Back (L-R): James Bach, Aaron Hodder, Andrew Robins, Bruce Graham, Tad Dawidoski, Dion Woodbury, Jeff Bedwell, Andrew Black, Oliver Erliwein, Farid Vaswani

Front (L-R): Brian Osman, Richard Robinson, Simon Crutchley, Nadine Brown, Sheryl Toenders, Chris Stapleton, Tessa Benzie

The reasons why are this:-

• James is more than an entertaining teacher – HE challenges you, your thinking, your ideas and THEN when you are confused, he explains what has happened (socratic method).
• The material covers more than a lot of other courses cover.Other courses will talk about techniques or numbers or data but they don’t connect the human side to the technical side. This is important because understanding how people engage, work, play, think IS just a part of testing as knowing the domain that you’re working in.

This is one of a very small handful of courses where i was ENGAGED througout the entire time. My brain was full and buzzing after every day!

James tester star

 I did manage to earn a tester star on the last exercise of the day. I won’t give the exercise away rather it felt like 2 1/2 days of thoughts and ideas came flowing out in that one exercise. It was easy to see how productive i was in that exercise as opposed to writing a script based on some requirements.

One is engaging, the other not so.

Learnings

 Two key things i have taken away is:

• Thread based test management (TBTM). From the last exercise of day 3, James pointed out that the work i was doing is essentially thread based. TBTM is an activity based approach to testing and i like how that works so i will be exploring that *thread* further (see http://www.satisfice.com/blog/archives/503)
• And the other is thinking, using and defending my approach to testing – that context driven, heursitical based approach is plausible, creative, detailed, formal and a more productive way to test. In a way, it reminds me of a sentence from Jerry Weinberg’s book – Perfect software and other illusions - “You fall victim to equating information quality with data quanity” – we equate document heavy processes and traditional thinking towards testing to be more effective than engaging your brain. There are better and more productive ways to approach testing

More to follow…

Yesterday a member of the group *ranted* (his words) about an *approved* job add that was posted.

It has spawned an interesting discussion on what it takes for a tester to get a foot in the door and has morphed into learning about the industry/certification.

I consider the discussions pure gold – check it out, comment if you wish – i think it would be helpful to hear about other testers thoughts/ideas/experiences from around the world!

 Aggressive testing, to me, is the art of asking the product questions, to think outside of the box (and the text-book) and to try different ways to test the application looking for interesting information (whether they be bugs, issues or curios). I prefer to be an aggressive tester. My mindset is to look for ways in which a product could fail. I see this as our value add as testers. When we find a bug, the bug is reported and resolved in *someway* thereby helping increase the quality of the product.

I believe that, while there is an element of passive testing (and what i mean here is checking), a tester is more beneficial to a project IF they are being aggressive and proactive and looking for potential failures or issues.

Detailed testing scripting can be *aggressive* in some ways but I’ve found that by having a pre-determined course of action, I am more likely to allow confirmation bias to influence the way I work. By exploring (and I mean having some structure – whether it is by session based test management or using high level test conditions/risks/ideas), I have found that I am more likely to be more aggressive in nature and pursue lurking bugs in the code as I have not been constrained by following detailed test steps.

I believe that to be an effective tester ultimately means that we are aware of the context of the project, application and environment, we are in pursuit of information (bugs, issues or curios - curio *term* taken from a discussion on twitter from James Bach and Michael Bolton) and we are feeding this information back into the project thereby helping management make more of an informed go/no go decision.

In one project I worked on there was a significant element of what i call Passive testing which came in the form of *running* a regression suite. This involved executing a test script which quickly devolved in an almost meaningless tick off and check exercise (check the test step – is it correct? – If so, check it off, if not do a superficial investigation (though I can state that no regression bugs were *found* as a result of this *testing*!)

This is bad passive testing which unfortunately is common in my part of the testing world. How many times have I seen disengaged *testers* running scripts that supposedly are meant to add value to the project – to my sceptical mind, all they add is paper.

Now, not all passive testing is as I’ve described. Even when we are aggressive in our approach there may still be elements of passive testing (checking against rules, contracts, laws, configurations, environments or anything that may require checks that help support what we do as engaged testers).

Aggressive and passive testing are NOT mutually exclusive – they are interdependent and intertwined – the issue I have is when the passive (non-engaged) testing is more prevalent than the engaged (by being engaged I mean brain switched on testing).

Unfortunately this is common in bigger organisations where I live. Fortunately, there are pockets of very engaged, context driven testers around that add much more value than the stock standard factory schoolers. I am thankful for that for it means that I’m not a lone voice in this part of the world!

New terminology?

Am i introducing a new set of terminology? No. Will others view testing as aggressive or passive? Maybe not. What I am doing is highlighting how I see testing. There is nothing wrong with that. I am not constrained by a glossary (though they could be useful) rather I am attempting to demonstrate what i mean by testing and how I view the craft that I work in.

Check it out!

If you are interested, please send your cv through to Tim Moore

To register for the webinar click here.

Also check here to see how New Zealand time compares with the time in your part of the world.

Look forward to having you tune in!