The one minute speed dribble syndrome

Rob Rangi is a very good friend of mine who happens to coach the St Mary’s Senior Girls Basketball team based in Wellington, New Zealand. He is blogging about his coaching experiences here.

He recently blogged about a recent session entitled Taking the Positives from the failures of drills. Coach Rangi is installing the Read and React offense, an offense that is based around principles rather than set plays.

Unlike a set play where, for example, player O1 passes to player O2 after player O2 was screened by player O3 (i.e. a structured offensive set), the Read and React is based on a group of principles in which the offensive players move depending on what is happening. This leads to an infinite number of possibliities in which the offense  can move, react and score. There is no blinked eyed approach whereby player O1 must do this in order to satisfy the pattern and potentially miss a scoring opportunity.

To quote from Coach Rick Tolbert (the Read and React creator), “…And that’s exactly what the Read and React Offense does: it provides a framework that can be used as an offensive system to develop players, teams, and programs. Or, it can be an offense for one team, an offense that builds upon itself, with a counter for anything any defense can throw at it.” Notice that Coach Tolbert talks about a framework. There is no mention of the words structured, pattern or set. In essence, the framework provides the heuristics (and the principles are collectively the oracle), the players apply these heuristics and adapt them  during game time.

Coach Tolbert also went on to stat his past season and found that 80% of his teams points came from principled basketball. Only 20% came from set plays and yet in practise, his team set spent 80% of the time on only 20% of the total point production!

Exploratory Testing is like the Read and React offense. It allows a creative (heuristics based), flexible (adaptable) approach (principles) to software testing that enables a tester to test a product with a broader mindset.

On the other side of the coin, writing test scripts (or if you like, using set plays)  is a very common testing practise which enables the tester to set out in advance, the steps he or she will follow.

One of the dangers of following a script is that the tester becomes a verifier of the steps as opposed to finding bugs or flaws or issues within the product.

And yet isn’t finding bugs the goal of testing?

Finding bugs is the value add testers bring to a project because by finding bugs and getting them fixed, the project team begin to increase the reliability of the system and potentially the quality as well.
This is nothing new. Glenford Meyers in his 1979 book ‘The Art of Software Testing’talks about his definition of testing

 “Testing is the process of executing a program with the intent of finding errors.”

It is not saying that testing should ensure that the product performs as specified or some such similar activity.

This is an important distinction – having the relevant mindset will steer us in the relevant direction. If we are looking to confirm that the product meets the specifications then it is likely that we can do this but will miss bugs. If, however, we are looking for bugs then we will find them (and along the way we will have false alarms or ‘non-bugs’ but isn’t that potentially better than missing some important bugs?).

Professor cem Kaner (Florida Institute of Technology) talks about this in the course Bug Advovacy  and also in his slide set that extends on his book Testing Computer Software. Prof. Kaner refers to what is called Signal Detection Theory. SDT quantifies the ability to discern between signal and noise and is a way in which psychologists measure the way decisions are made under conditions of uncertainity. When we are testing, there is nothing more uncertain as software we are have been just been given!

This of course can be influenced by the rules or limits or bias we set on ourselves or the group of testers we look after. Wikipedia has an excellent example of this bias

“Bias is the extent to which one response is more probable than another. That is, a receiver may be more likely to respond that a stimulus is present or more likely to respond that a stimulus is not present. Bias is independent of sensitivity. For example, if there is a penalty for either false alarms or misses, this may influence bias. If the stimulus is a bomber, then a miss (failing to detect the plane) may increase deaths, so a liberal bias is likely. In contrast, crying wolf (a false alarm) too often may make people less likely to respond, grounds for a conservative bias.”

In testing, if we influence testers to make sure the product conforms to requirements then we steer the bias in that direction. If we influence the bias towards finding bugs then that is what will happen and as Glenford Meyers has already pointed out, we begin to add value (potentially at a greater add than if we are looking to confirm that the product meets requirements).

Coach Rangi struck an interesting dilema at one practise. He asked his team to run a full court drill involving the speed dribble and read and react principles. This is what happened…

Coach : “OK Ladies we’re going to do a minute using the Speed dribble. Read the ball and react accordingly”

Players : “Yes Coach!”

Point Guard brings the ball to the top from our 2-man fast break. Our Wing running the outside lane, get her wing position and almost without hesitation cuts to the basket. So I stop the drill and pull her up.

Coach : “OK, What was your Read?”
Player : “Ah that was the speed dribble coach”
Coach : “OK So you made the cut although X actually hadn’t started the speed
dribble towards you”
Player : “Yeah, I was anticipating her doing the speed dribble at me”
Coach : “Why would you be anticipating it? You should be reacting to what she
does? What would happen if she drove or wanted to make a pass?”
Player : “But she wouldn’t do that Coach”
Coach : “And why is that?”
Player : “Cause you said we were running Speed Dribbles for a minute”

What an interesting sequence! Look at how Coach sets or influences the drill’s bias (just like following a script). Then the team interprets his instructions and follows the “script” to achieve the aim of the drill (“OK Ladies we’re going to do a minute using the Speed dribble. Read the ball and react accordingly”). The player then inteprets the instruction without question and becomes inflexible and doesn’t adapt to what the point guard was doing.

Coach Rangi then went on to say…
“…So after practice, I reviewed our training and was able to determine that the drills suffered from having a pre-conceived outcome based on a known condition eg we’re doing pass and cut for a minute then speed dribble for a minute then natural pitch etc. We needed to remove the pre-conception and make it random forcing the Wing to work.”

Fantastic! Much like in software testing where we have an expected result based on a known condition, our ability and effectiveness to analyse, think critcally and discover bugs is reduced by the bias surrounding our testing (test scripts or in basketball, set plays). We can become almost paralysed by following and completing each step in the script (been there, done that) and lose potential ideas, thoughts and creative ways in which to discover bugs (i have personally experienced both mindsets probably as most testers have at one stage or another).

How then did Coach Rangi fix this…

“We now have a new drill called “You make it up 2-man break”. We run 2 minutes using Circle movement options only – Dribbler drives, Dribbler drives and pitches, Dribbler drives and pivot pass to Safety valve. Then we run another 2 minutes using the other options – Pass and Cut, the Speed and Power Dribble. We also instigated a rule that says the next pair to go cannot do the same move as the pair in front has just done ensuring a different option each time down the court.”
Coach Rangi then finishes his blog by saying…

“In hindsight I should’ve seen this coming but there is nothing like getting it on the floor and letting players find the flaws for you. And honestly, I’m glad they did because it just made us a better basketball team!”

Much like in software testing, Exploratory testing is an approach that can help us become alot more flexible and help us avoid the “Cause you said we were running Speed Dribbles for a minute” syndrome!

Can’t Beat Experience!

It has been awhile since I’ve written and its mainly because i have moved into the “academic” side of testing – delivering software testing courses for Software Education in New Zealand. As a result i have been busy travelling and delivering!

One of the main things that I’ve noticed during the course delivery is  the degree of separation between a junior test analyst and someone with more experience. At the end of the day, it appears to come down to how many more stories or experiences that someone who has been in the *game* longer has to draw on. This lead me to think about a quote from W.Edward Deming:-

“Experience by itself teaches nothing.” This statement emphasizes the need to interpret and apply information against a theory or framework of concepts that is the basis for knowledge about a system. It is considered as a contrast to the old statement, “Experience is the best teacher” (Dr. Deming disagreed with that). To Dr. Deming, knowledge is best taught by a master who explains the overall system through which experience is judged; experience, without understanding the underlying system, is just raw data that can be misinterpreted against a flawed theory of reality. Deming’s view of experience is related to Shewhart’s concept, “Data has no meaning apart from its context” (see Walter A. Shewhart, “Later Work”). – http://en.wikipedia.org/wiki/W._Edwards_Deming

From my perspective, the more “battles” one has been in, the more experiences to draw from (even if one has “only” been in one organisation) and to some extent, the approach of the test analyst (Agile, Automated, Exploratory, Scripted, UAT etc) may not been as relevant as it serves to broaden the range of ones knowledge.

When i first started testing, one of the theories that i held/learnt was that testing breaks software. When i was asked what i do, i would often respond “I test  software by breaking it”.

Over time, associating with different projects and talking to many people, i view testing as Archeology – we sift through the dust and dirt using whatever approach is necessary to uncover the bugs – the software is already broken – we look for clues to find where these may hide!

Therefore, as testers, it is up to us to find our theory – whatever or wherever that may be. Seek to learn, broaden your skills and then apply this theory when getting your hands *dirty* . This will help to broaden our experience and maybe, just maybe, our individual value as a tester!

BTW – I’m currently reading an Introduction to General Systems Thinking by Gerald M Weinberg – so I’m beginning to walk the talk!

Happy learning!

An Expression of Thought – Testing Ideas

Having no way as way, having no limitation as limitationI have become a fulltime trainer working for Software Education in New Zealand ( www.softed.com ) delivering software testing courses. As i’m now sitting more in the Academic space as opposed to the Practioner space, I have been given the opportunity to meet many different people, with different backgrounds, looking to gather new ideas to use in their testing jobs.

Some people won’t do much, if anything with this new knowledge (it’s human nature after all especially when the work pressure comes on) but some will. It is these testers that will hopefully feel inspired to share their thoughts and ideas with us all.

The internet has made us a very small, very connected global community and each thought expressed or shared (particularly in testing) is a thought worth considering. Maybe you have discovered a new idea with regards to testing or maybe even reaffirming an existing idea (and adding your own wrapper around it).

To those testers that i have met or to those people who may be reading this and haven’t yet considered creating a blog, please rethink. Your thoughts are valuable and your ideas are at least worthy of expression and/or comment.

I would to *hear* them – please let me know if you do! 

Happy blogging!

Exhaustive Testing

exhausted.jpg

 The following is  a response i sent to Kit who commented on my blog on ‘Insufficient Testing’ ….

Thanks for your comment. It’s almost a catch 22 situation. One of the principles of testing (according to ISTQB) is that Exhaustive Testing is impossible – I agree but the question is how much do you test and when do you know enough is enough?

For a complex system my thoughts would center around risk and priorities as your starting point. The approach or method used would ultimately rest on what level of auditability you must provide to the Business (they ultimately make the decision to go or no go.) Personally I would still use Exploratory Testing (if I was ‘allowed’ to) because in my experience I would be more likely to find something of value more often than through scripts.

However, in saying that, if the test team is involved right at the beginning of the project through walkthroughs, reviews or inspections (or any other type of review)than clarification and understanding will no doubt increase amongst the testing team with regards to the system.

After doing a Wikipedia search on Dr. Deming, one of his quotes is quite applicable to software testing… “Acceptable Defects: Rather than waste efforts on zero-defect goals, Dr. Deming stressed the importance of establishing a level of variation, or anomalies, acceptable to the recipient (or customer) in the next phase of a process. Often, some defects are quite acceptable, and efforts to remove all defects would be an excessive waste of time and money.” It is known that major commercial software often ships with known (and unknown) defects – MS Windows, Firefox v2.0 etc – its is reasonable then for the business to decide how much of the ‘risk’ they wish to carry. Testers should provide the necessary information to enable business to make that decision (good or bad).

At one New Zealand bank that I worked in, the test team I became involved with tried hard to exhaustively tested everything in a very complex application. The upshot was that one release took almost 12 months to ‘complete’ testing (there were other factors involved – personnel, political and management)BUT I guarantee that they could not say that that application was bug free. So I guess that leads to the second question – how much is enough?

James Bach says “When I exhausted the concerns of my internal critic (and external critics I asked to review my work), I decided it was good enough” (refer http://www.satisfice.com/articles/how_much.shtml).

NASA’s software safety standard (http://satc.gsfc.nasa.gov/assure/nss8719_13.html) NASA-STD-8719.13A September 15, 1997 – Section 3.4.5 says “The test results shall be analyzed to verify that all safety requirements have been satisfied. The analysis shall also verify that all identified hazards have been eliminated or controlled to an acceptable level of risk. The results of the test safety analysis shall be provided to the ongoing system safety analysis activity.” What then is an acceptable level of risk and acceptable to whom? Risk is then defined in this document as “…As it applies to safety, exposure to the chance of injury or loss. It is a function of the possible frequency of occurrence of the undesired event, of the potential severity of resulting consequences, and of the uncertainties associated with the frequency and severity.” Also in the document under section 1.4 Tailoring it says “….The tailoring effort shall include definition of the acceptable level of risk, which software is to be considered safety-critical, and whether the level of safety risk associated with the software requires formal safety certification.” Therefore at the end of the day , it’s a business decision taken within context of the business. As testers, we can test complexity within the context of the project and report back our findings – it is then up to those charged with making the ‘big’ decisions, to make them – or not!

Insufficient Testing

F-22 RaptorsIs a test team ‘liable’ if the product/software fails in some way? A recent post to the Software Testing Yahoo groups forum brought this to light and got me thinking.

Jared Quinert – a proponent of ET from Australia said “…a lack of testing – that insufficient testing requires some co-conspirator to cause a project to fail?
Sadly, nothing stops people trying. Googling ‘”insufficient testing” project failure’ goes some way to demonstrating this.”

So i did….try googling “insufficient testing” and see what comes up. There are, according to Google, 493,000 references to insufficient testing. This then begs the question – What is insufficient testing?

I worked recently within a test group that was fixated on exhaustive testing – they literally wanted to test everything and anything (and with good reason i might add – the situation i.e. context – surrounding them was NOT conducive to a co-operative approach. The harder the test group tried the more they got blamed.) It was hard to changed that mindset because they had litteraly been burnt in the past. What this meant was a huge overhead in terms of time. This group is the opposite of insufficient testing because they wanted to do everything.

However, it is a fact of life (this has been well documented in a number of articles, blogs etc) that software testers cannot find everything. Software is complex (ask NASA), software can be daunting and despite testing things do go wrong – just ask the US Air Force

(http://en.wikipedia.org/wiki/F-22_Raptor#Recent_developments )

“While attempting its first overseas deployment to the Kadena Air Base in Okinawa, Japan, on 11 February 2007, a group of six Raptors flying from Hickam AFB experienced multiple computer crashes coincident with their crossing of the 180th meridian of longitude (the International Date Line). The computer failures included at least navigation (completely lost) and communication. The fighters were able to return to Hawaii by following their tankers in good weather. The error was fixed within 48 hours and the F-22s continued their journey to Kadena”

Was this fault because of insuffcient testing or was it the result of other factors? In my experience of failed projects, insufficient testing usually isn’t the cause rather a lack of cohesion between PM, vendor, BA’s, developers, testers – each group assumed a territorial stance and placed their ego in the way.

As Gen. Colin Powell (ret) says ” never let your ego get so close to your position that when your position falls, your ego goes with it.”

Often there was some sort of conflict or barrier (whether declared or otherwise) that existed in which the leadership group was unable to break through. Disharmony in a project team will definitely achieve less with more.

So then is insufficient testing clearly a fault of the test team?

 Sometimes it is.

If the team was not aligned to the Project goals and was off on their own agenda then yes. However, if there are external influences involved then insufficient testing may be a symptom of a bigger problem.

Testing the Mindset

tao_yinyangearth2.jpgI recently read an interesting blog entitled ‘How can i become a better Tester’ – http://thoughtsonqa.blogspot.com/2007/12/how-can-i-become-better-tester.html

This was my comment i left… 

Hi John,

Enjoyed your article. I agree – its mindset (quality), its information gathering (read, read and more reading….asking questions…be involved)and finding that mentor who you can clicked with. Sometimes, when as a new tester, we can be blinded by the bias of that mentor so i would add – ‘When you are ‘ready’ question yourself, your understanding, your toolbox and then define yourself in the testing space’ – the trick is knowing when you are ready!
When i first started testing i was sure that testing was <b>ALL</b> about test scripts, test documents, writing documents and more documents because that’s how it was. Today, my thoughts and process have changed dramtically compared to when i first started testing but those earlier experiences shaped my thought processes today!
Great blog John!

Which got me thinking – how do our experiences shape our thought processes and ‘steer’ us towards one method or another? For me embracing a more Exploratory approach was a logical evolution in the testing space. It allowed me to be creative yet structured at the sametime – it increased my toolbox – and i gain immense satisfaction from this approach to testing. Why? Because when i was involved in the more traditional form of testing, i got to the point that i wondered what is the point to what i am doing….in other words i began to question myself and re-examined the ‘tools’ i had. That’s when i became open to different methods to testing.

If i wasn’t as receptive or i wasn’t at that questioning stage, i doubt that Exploratory testing would’ve taken off for me as it has!

So sometimes, it comes down to timing as well as being open to new ideas!

Teamwork

teamwork-skydivers-ii-print-c10007532.jpgI was reading a book from master coach Pat Riley entitled The Winner Within. This book has been around for a number of years (1993) and Coach Riley discuss his philosophies that make up a successful team. Coach Riley knows what he’s talking about – he’s was at the helm of the 1980’s LA Lakers World Championship basketball teams, the New York Knicks and the 2006 Miami Heat championship team.

Most of us belong to some sort of team. And while we may not always get on with others, we are some what reliant on others doing their job, playing their role. Its no different from testing. Whilst we may be perceived as being negative (‘Your job is to break the system’), we still play a vital part. If we then are able to look at the big picture and synergise with the (project) team as a whole then we are able to produce quite effecient results.

You see this all the time in sports where a very talented team just can’t seem to get it together. I once coached a basketball team that, individually, were quite brilliant for their age group. But as a team they just couldn’t take the ‘I’ out of team. A championship team (or for that matter a good team) is a team that is able to synergise well together – their is no ego only healthy respect for each other. BA’s working with Testers and developers in harmony to produce something extraordinary (one government department i worked for worked exactly like that and yet another didn’t because there was a wall between the testers, BA’s and developers – literally and figuratively).

One of the ways to build this teamwork is founded on trust. It drives us, it helps us, it builds confidence in each other and in others. It enhances who we are and yet at the sametime if we trust and are trusted then we are more likely to ‘build up’ then tear down. A divided house cannot stand.

As Benjamin Franklin states on the signing of the Declaration of Independence “We must all hang together, else we shall all hang seperately”

Test Strategy vs. Test Plan

sun_tzu.jpgRecently i have posted a couple of replys on the Microsoft Software Test forum – http://msdn2.microsoft.com/en-us/testing/default.aspx – click the link entitled Software Testing Discussion Forum.

There are a couple of posts on Test Strategy and Test Plans and what they are. Quite often the two terms are interchangeable and used indiscriminately. A Test Strategy (according to ISTQB) is the document that describes the methods of testing or the how. Whether this document is pitched at an enterprise level or a project level is open to discussion but essentially the Strategy is projecting ideas over a longer period of time.

The Oxford Dictionary defines Strategy as “… a plan designed to achieve a particular long-term aim” and as such looks at the ‘bigger’ picture.

A Test Plan describes the ‘How’ at a lower level. IEEE 829 (on which there is much debate on the revision currently being voted on) introduces the structure of should be incorporated in the plan. It is comparable to tactics which again according the Oxford Dictionary is “…an action or strategy planned to achieve a specific end.”

Whether you use either or both terms or documents is up to you. As testers we sometimes become involved in paper wars and become document heavy at the expense of efficiency and effectiveness. Whatever process you follow the key for any test document is effective communication.

Bj Rollison – a Test Architect at Microsoft (http://blogs.msdn.com/imtesty/about.aspx) sums up what happens if we stop thinking about how and what we use these documents for…”The only testers who stop thinking critically about tools and the application of tools we can use in the appropriate context are testers who have a limited understanding of the overall responsibility of testing, and know even less about the tools they are tyring to use.”

Great quote – i totally agree. Whether its a Test Strategy or Test Plan, it is a tool whose purpose is to serve us and guide us in our testing activities.

If you are responsible for producing said documents, please be critcial in your thinking and look at the best way to communicate to all those involved in your sphere!

The 2006 NBA season the Phoenix Suns Strategy vs. the LA Lakers was “…Phoenix’s strategy against the Lakers this season has been to contain everyone besides Kobe Bryant, and it’s worked. Bryant had 39 points in the first meeting and 37 in the second, but no other Lakers player scored more than 17 in the two games and the Suns simply outscored Los Angeles, averaging 114 points.” – Yahoo Sports.

There is your Strategy, the plan is how each Sun player played defense against their man.

Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat.– Sun Tzu