Agile @ StarWest Software Testing Conference 2012

In my experience, what makes agile so powerful is the  encouragement of rapid, effective communication to achieve, uncover and discover what is wanted, what is being built and what could be going wrong. Practices such as collaboration and co-location can be effective tools for any project regardless of whether your project is agile or not.

I will be at Star West 2012, Anaheim, California next week (1 October 2012 – 5 October 2012) and will be speaking Thursday on how I borrowed some agile practices for a non-agile project and the lessons that I learnt.

If you’re attending Star West, come a long and say hello otherwise I will blog and tweet where I can!

See http://www.sqe.com/StarWest/Concurrent/Default.aspx?Date=10/4/2012#T19

Whom shall I serve?

Tweet - Mike Talks
Tweet – Mike Talks – KWST#2 – June 2012

Whom shall i serve?

A song, a hymn, or a reminder as to who our customers are? Who do we serve and why is that important?

During KWST#2 (June 2012, Wellington, New Zealand), the discussion about whom  we serve came up.

Mostly, the answers tended to support the obvious conclusion (to me at least) that whom we serve could be:-

Our employer(s)
The project manager
The developers
The business
The test manager
The test team
The project team
Our family

And these are all valid customers/people/organisations/groups that we give service to in some way. But there is one other element that sometimes we don’t consider…

Ourselves

Whom shall I serve? I think first and foremost it is ourselves. We are responsible for our own work, for our own ethics, our output, our own learning, our own interactions with others, our own interactions with other testers and our own interactions with the software testing community.

Sometimes we take a high degree of responsibility for one or some of these things and sometimes we don’t. What may be important is that we come to understand that we also serve ourselves and by seeing ourselves as a customer (if you will) then it allows us to appreciate who we are as a tester, what we can deliver, what skills we have and what we stand for.

Too often I have seen testers wilt in the face of criticism (and scrutiny for that matter) from management attempting to justify testing or test artifacts or activities. Knowing what we stand for gives us a moral ground to argue from. Unfortunately, it doesn’t mean that everything will be *perfect* because we are conscious of our position but at least we know our tipping point.

So how do you deal when reaching your tipping point?

Well, that does depend but some of the ways that I have used have been:-

  • Educate those that may be pushing you towards your tipping point – (in my experience, it is typically a manager)
  • Listen to those pushing you to your tipping point – (it is possible that we don’t understand their context)
  • Use your influence and credibility to help educate
  • Employ a stealth approach – (one project I was on, the project wanted test cases (with expected and actual results) and use what they saw as structured testing. While we spent time giving them what they wanted, the majority of the issues during test execution came, not from the test cases, but from an undeclared exploratory approach. OUR plan of attack became give the customer what they wanted, educate them along the way and use good exploratory testing to find valuable information *quickly*. The test cases in this instance were our checks, the exploratory test charters, our tests. The stealth here was from discerning the clients context,employing what became a blended approach and not necessarily letting management know that this is what was happening.)
  • Leave – (this is most likely the extreme option but sometimes it is more beneficial to/for you to leave a project/employer/organisation than having to adhere to rules that may not make sense. I have done this, it was a challenge but I’m glad I did it.)

So, whom do we serve? Ourselves first (it’s not as selfish as it may seem) and then those mentioned above. Putting ourselves first means that we are taking responsibility for the quality of our own work which means in turn, we are better placed to serve our customers.

The one minute speed dribble syndrome

Rob Rangi is a very good friend of mine who happens to coach the St Mary’s Senior Girls Basketball team based in Wellington, New Zealand. He is blogging about his coaching experiences here.

He recently blogged about a recent session entitled Taking the Positives from the failures of drills. Coach Rangi is installing the Read and React offense, an offense that is based around principles rather than set plays.

Unlike a set play where, for example, player O1 passes to player O2 after player O2 was screened by player O3 (i.e. a structured offensive set), the Read and React is based on a group of principles in which the offensive players move depending on what is happening. This leads to an infinite number of possibliities in which the offense  can move, react and score. There is no blinked eyed approach whereby player O1 must do this in order to satisfy the pattern and potentially miss a scoring opportunity.

To quote from Coach Rick Tolbert (the Read and React creator), “…And that’s exactly what the Read and React Offense does: it provides a framework that can be used as an offensive system to develop players, teams, and programs. Or, it can be an offense for one team, an offense that builds upon itself, with a counter for anything any defense can throw at it.” Notice that Coach Tolbert talks about a framework. There is no mention of the words structured, pattern or set. In essence, the framework provides the heuristics (and the principles are collectively the oracle), the players apply these heuristics and adapt them  during game time.

Coach Tolbert also went on to stat his past season and found that 80% of his teams points came from principled basketball. Only 20% came from set plays and yet in practise, his team set spent 80% of the time on only 20% of the total point production!

Exploratory Testing is like the Read and React offense. It allows a creative (heuristics based), flexible (adaptable) approach (principles) to software testing that enables a tester to test a product with a broader mindset.

On the other side of the coin, writing test scripts (or if you like, using set plays)  is a very common testing practise which enables the tester to set out in advance, the steps he or she will follow.

One of the dangers of following a script is that the tester becomes a verifier of the steps as opposed to finding bugs or flaws or issues within the product.

And yet isn’t finding bugs the goal of testing?

Finding bugs is the value add testers bring to a project because by finding bugs and getting them fixed, the project team begin to increase the reliability of the system and potentially the quality as well.
This is nothing new. Glenford Meyers in his 1979 book ‘The Art of Software Testing’talks about his definition of testing

 “Testing is the process of executing a program with the intent of finding errors.”

It is not saying that testing should ensure that the product performs as specified or some such similar activity.

This is an important distinction – having the relevant mindset will steer us in the relevant direction. If we are looking to confirm that the product meets the specifications then it is likely that we can do this but will miss bugs. If, however, we are looking for bugs then we will find them (and along the way we will have false alarms or ‘non-bugs’ but isn’t that potentially better than missing some important bugs?).

Professor cem Kaner (Florida Institute of Technology) talks about this in the course Bug Advovacy  and also in his slide set that extends on his book Testing Computer Software. Prof. Kaner refers to what is called Signal Detection Theory. SDT quantifies the ability to discern between signal and noise and is a way in which psychologists measure the way decisions are made under conditions of uncertainity. When we are testing, there is nothing more uncertain as software we are have been just been given!

This of course can be influenced by the rules or limits or bias we set on ourselves or the group of testers we look after. Wikipedia has an excellent example of this bias

“Bias is the extent to which one response is more probable than another. That is, a receiver may be more likely to respond that a stimulus is present or more likely to respond that a stimulus is not present. Bias is independent of sensitivity. For example, if there is a penalty for either false alarms or misses, this may influence bias. If the stimulus is a bomber, then a miss (failing to detect the plane) may increase deaths, so a liberal bias is likely. In contrast, crying wolf (a false alarm) too often may make people less likely to respond, grounds for a conservative bias.”

In testing, if we influence testers to make sure the product conforms to requirements then we steer the bias in that direction. If we influence the bias towards finding bugs then that is what will happen and as Glenford Meyers has already pointed out, we begin to add value (potentially at a greater add than if we are looking to confirm that the product meets requirements).

Coach Rangi struck an interesting dilema at one practise. He asked his team to run a full court drill involving the speed dribble and read and react principles. This is what happened…

Coach : “OK Ladies we’re going to do a minute using the Speed dribble. Read the ball and react accordingly”

Players : “Yes Coach!”

Point Guard brings the ball to the top from our 2-man fast break. Our Wing running the outside lane, get her wing position and almost without hesitation cuts to the basket. So I stop the drill and pull her up.

Coach : “OK, What was your Read?”
Player : “Ah that was the speed dribble coach”
Coach : “OK So you made the cut although X actually hadn’t started the speed
dribble towards you”
Player : “Yeah, I was anticipating her doing the speed dribble at me”
Coach : “Why would you be anticipating it? You should be reacting to what she
does? What would happen if she drove or wanted to make a pass?”
Player : “But she wouldn’t do that Coach”
Coach : “And why is that?”
Player : “Cause you said we were running Speed Dribbles for a minute”

What an interesting sequence! Look at how Coach sets or influences the drill’s bias (just like following a script). Then the team interprets his instructions and follows the “script” to achieve the aim of the drill (“OK Ladies we’re going to do a minute using the Speed dribble. Read the ball and react accordingly”). The player then inteprets the instruction without question and becomes inflexible and doesn’t adapt to what the point guard was doing.

Coach Rangi then went on to say…
“…So after practice, I reviewed our training and was able to determine that the drills suffered from having a pre-conceived outcome based on a known condition eg we’re doing pass and cut for a minute then speed dribble for a minute then natural pitch etc. We needed to remove the pre-conception and make it random forcing the Wing to work.”

Fantastic! Much like in software testing where we have an expected result based on a known condition, our ability and effectiveness to analyse, think critcally and discover bugs is reduced by the bias surrounding our testing (test scripts or in basketball, set plays). We can become almost paralysed by following and completing each step in the script (been there, done that) and lose potential ideas, thoughts and creative ways in which to discover bugs (i have personally experienced both mindsets probably as most testers have at one stage or another).

How then did Coach Rangi fix this…

“We now have a new drill called “You make it up 2-man break”. We run 2 minutes using Circle movement options only – Dribbler drives, Dribbler drives and pitches, Dribbler drives and pivot pass to Safety valve. Then we run another 2 minutes using the other options – Pass and Cut, the Speed and Power Dribble. We also instigated a rule that says the next pair to go cannot do the same move as the pair in front has just done ensuring a different option each time down the court.”
Coach Rangi then finishes his blog by saying…

“In hindsight I should’ve seen this coming but there is nothing like getting it on the floor and letting players find the flaws for you. And honestly, I’m glad they did because it just made us a better basketball team!”

Much like in software testing, Exploratory testing is an approach that can help us become alot more flexible and help us avoid the “Cause you said we were running Speed Dribbles for a minute” syndrome!

The Pursuit of the Unreplicable Bug

ghostbusters.gifI’ve been recently testing a web-based application that produced a very interesting defect. It seemed that in one particular screen, a user (with the right combination key strokes and mouse clicks) could actually enter a supposedly uneditable error message field and enter text! At first i wasn’t able to repeat this behaviour but with the words from a James Bach article ringing in my ears about “…ignoring unreproducible bugs at your peril”, i logged it waiting for the right opportunity to attack it.

I had already spent time looking for this ‘bug’ but figured that i would put it to one side and come back to it with fresh eyes and clearer thoughts. Interestingly enough, the developers caught hold of this bug and attempt to replicate in their dev environments – i was even ‘challenged’ in a joking way that if i couldn’t reproduce the bug within 5 attempts then it didn’t exisit!! Oh, did the competitive urges come out then! (This was done in good spirits – we have a tremendous rapport between developers, testers and BA’s). However, it was another developer that found the key/mouse strokes that generated the bug and we discovered that it was a validation error on that web page!

So what were the lessons learnt?

  1. Exploratory testing found this bug – some may say that discovery was a ‘fluke’ but scripted testing would never have picked this bug up.
  2. Fresh eyes and a clearer head can aid tremendously in trying to replicate a bug (especially one discovered late in the day!)
  3. Having a rapport with developers helps in solving bugs – personal agendas and politics are put to one side for the greater good of the ‘team’
  4. Working alongside developers generally breaks down communication barriers (percieved and physical)
  5. Unreproducable bugs ARE best ignored at ones own peril – in this case finding this bug lead to a tightening of field validation for the application
  6. Bugs are bugs are bugs…testers find them, developers fix them, buisness decide what they want done with them – never give up on trying to replicate bugs that are difficult to reproduce!
  7. Teamwork – i honestly believe the power of many can be greater than the power of one
  8. It’s tremendously satisfying finding a bug that is difficult to find and reproduce – the testing equilivant of a three-pointer!