29 November, 2007 at 10:17 pm (Exploratory Testing, Software Testing, testing)
Tags: Exploratory Testing, Software Testing, Test Management, Test Plan, Test Strategy
Recently i have posted a couple of replys on the Microsoft Software Test forum – http://msdn2.microsoft.com/en-us/testing/default.aspx - click the link entitled Software Testing Discussion Forum.
There are a couple of posts on Test Strategy and Test Plans and what they are. Quite often the two terms are interchangeable and used indiscriminately. A Test Strategy (according to ISTQB) is the document that describes the methods of testing or the how. Whether this document is pitched at an enterprise level or a project level is open to discussion but essentially the Strategy is projecting ideas over a longer period of time.
The Oxford Dictionary defines Strategy as “… a plan designed to achieve a particular long-term aim” and as such looks at the ‘bigger’ picture.
A Test Plan describes the ‘How’ at a lower level. IEEE 829 (on which there is much debate on the revision currently being voted on) introduces the structure of should be incorporated in the plan. It is comparable to tactics which again according the Oxford Dictionary is “…an action or strategy planned to achieve a specific end.”
Whether you use either or both terms or documents is up to you. As testers we sometimes become involved in paper wars and become document heavy at the expense of efficiency and effectiveness. Whatever process you follow the key for any test document is effective communication.
Bj Rollison – a Test Architect at Microsoft (http://blogs.msdn.com/imtesty/about.aspx) sums up what happens if we stop thinking about how and what we use these documents for…”The only testers who stop thinking critically about tools and the application of tools we can use in the appropriate context are testers who have a limited understanding of the overall responsibility of testing, and know even less about the tools they are tyring to use.”
Great quote – i totally agree. Whether its a Test Strategy or Test Plan, it is a tool whose purpose is to serve us and guide us in our testing activities.
If you are responsible for producing said documents, please be critcial in your thinking and look at the best way to communicate to all those involved in your sphere!
The 2006 NBA season the Phoenix Suns Strategy vs. the LA Lakers was “…Phoenix’s strategy against the Lakers this season has been to contain everyone besides Kobe Bryant, and it’s worked. Bryant had 39 points in the first meeting and 37 in the second, but no other Lakers player scored more than 17 in the two games and the Suns simply outscored Los Angeles, averaging 114 points.” – Yahoo Sports.
There is your Strategy, the plan is how each Sun player played defense against their man.
Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat.- Sun Tzu
3 Comments
28 November, 2007 at 12:48 am (Software Testing)
The test plan is the framework for your testing activities. However, what is most important is the content not the actual format (whether its IEEE or an organisational format).
The test plan should outline what is being tested, how its being tested, what you do with artifacts generated during test and what happens if…. .If we become set on the ‘format’ then what we get is a process and document heavy testing activity that detracts from the actual testing!
How many times (i’ve seen many) have the TM or Lead spent hours, days, weeks on a plan, get feedback (which is usually minimal) and get it signed off only to have it resigned to the top drawer never to see the light of day!
If anything, the test plan should be a flexible document in the sense that the unexpected usually happens during testing. For example, what if one of your plan’s exit criteria says that testing is complete when 100% of the Test Scripts have been signed off but a blocking defect occurs that blocks 2% of those Scripts (and will not be fixed in this release). Does this mean the plan has failed? Were we wrong to make our Exit criteria so strict?Or does commonsense tell us that because the business have made the call to defer fixing the blocking defect to a future release then our test plan is still on track?
My point is that adherence to structure and format is secondary to the who, how, what, why and where’s. The structure gives a framework that we adapt to suit our situation (CONTEXT).
9 Comments
26 November, 2007 at 11:40 pm (Software Testing)
Observation-Orientation-Decision-Action (OODA) - Colonel John Boyd (USAF) was the theorist behind the OODA loop. Essentially, the quicker a fighter pilot can execute the OODA loop, the more of an advantage that person has over their adversary.
How can we relate this then to bug hunting? Well, simply put…
Observation - Observe our environment - what is happening when we spot a defect? Has it happened before, is it known? If its new, there may be other related bugs (bug clustering). Beware of anything that looks out of the ordinary or just doesn’t ‘feel right’.
Orientation – lets get our bearings – how did we get that bug? What were the steps? Is it easy or hard to reproduce? What are the triggers?
Decision – what are we going do with the bug? How do we prove our case to management and the developers? How severe is the defect? Are there any other bugs lurking around?
Action – Prove it that it exists, show that it exists and make it visible to those that are responsible for making the decision (Action) to do something about it!
The key is observation which can take many forms. If we are able to observe what we are testing (not just look at but in-depth observation) then we are able to draw out the necessary conclusions that allows us to satisfy our curiosity of the system at hand. With that curiosity we are then able to figure out where the bug appears, decide the appropriate way forward and act on it. Testing by nature is curiosity unleashed on a grander scale and with some sort of framework around.
What better job to do than satisfy ones curiosity?
Leave a Comment
25 November, 2007 at 10:10 pm (Software Testing)
I’ve just this morning read an interesting article on combinatorial or pairwise testing (http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf and http://shrinik.blogspot.com/2007/03/boundary-value-exploration-bve.html .) This article has been around for while but it covers a very valid point. Pairwise testing is Domain testing or as ISTQB call it Equivalence Partitioning. Equivalence Partitioning ”…is based on the premise that by testing one value within a class the result is representative of all values within that class.”
The technique, on the surface, has merit but how do we test for the unknown? What if our test data doesn’t cover that exception? Pairwise testing is a valid technique BUT as with all testing, it should be combined with the testers skill and judgement. To follow one technique for the sake of the technique is but to limit one’s effectiveness.
Judgement and skill are hallmarks of a good tester. Education and knowledge of the ‘craft’ are more important than the expertise in a particular system or application. There are many good subject matter experts but not alot of really good testers…testers that can think, change and adapt.
As Robert Sauborin likes to frequently quote in his course slides “No! Try not, Do. Or do not.There is no try.” – (Master Yoda instructing young Luke Skywalker in the ways of the force.)
In otherwords, look beyond what you do know…there may be another way to test your product…do it and evaluate for yourself. In is only after a period of exploration and experimentation that we can become aware of what the system can do.
Just because you (or the team) have always tested the system the same way doesn’t necesarily mean that it is the be all and end all of testing. Look outside the square and think!!
1 Comment
20 November, 2007 at 8:32 pm (Software Testing, testing)
Tags: Bruce Lee, exploratory, Fist, JKD, Software Testing, testing
Welcome to my first blog on software testing – hopefully one of many!
Testing is an exciting intellectually stimulating discipline to be involved in. I have been in the ‘game’ now for over 8 years in Wellington, New Zealand. I have worked in both the private and public sectors and over that time, my knowledge and understanding has been shaped by the experiences i have come across and the ‘teachings’ i have absorbed from other practitioners, organisations and learned colleagues around the world (Cem Kaner, James Bach, Brian Marick, Bret Pettichord among others). The combination of these things have shaped my view and understanding of testing as i know it today.
One of the greatest, if not the greatest martial artist of the 20th Century – Bruce Lee – espoused his philosophies on martial arts through a framework called ‘Jeet Kune Do’ (JKD) or Way of the Intercepting Fist.
One of the theories of JKD is that a fighter should do whatever is necessary to defend himself, regardless of where the techniques used come from. This is like testing – should we subscribe to only ‘one way’ or one technique to test software when there is a multitude of methods that may suffice? Should we follow the ‘classical mess’ (scripted testing) or become ‘formless like water’(Exploratory testing)?
Don’t get me wrong – test scripted pre-arranged testing has its place and it works – i have no qualms about that – I’ve been there, lived that. The only voice of dissent i have is when so called Test Practitioners disregard any other way of doing it. I worked once for a bank in which i prepared a paper on the benefits of exploratory testing – especially used in a complementary way with expected mode of scripted testing. It never gained traction no matter how much i tried – the Test Manager went as far as to say “…it doesn’t work and you’ll never find anything with that.”
I beg to differ.
As you can tell, i am an exponent of the Context Driven school and Exploratory testing. However, i know and recognise the need for scripting and i am comfortable with that. The trick is seeing what you test in context of what you are doing and what you are trying to achieve.
You may be familiar with the following – its applies whether you Exploratory test or not…
THE SEVEN BASIC PRINCIPLES OF THE CONTEXT-DRIVEN SCHOOL
1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project’s context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
Have a great week testing!
6 Comments
Recently i have posted a couple of replys on the Microsoft Software Test forum – http://msdn2.microsoft.com/en-us/testing/default.aspx - click the link entitled Software Testing Discussion Forum.
