Waterfall development has a drawback - it can be so functionally driven that testing of more abstract concepts such as “is the user interface any good?”, “can I migrate from version X to version Y” and “does my software handle failures?” can suffer. Why is this?

The core reason is that these concepts often span so many functional items, that they cannot be easily contained within testing of a single functional item. They tend to slot into the system verification test cycle - yet in the waterfall method this is at the end of the development cycle and probably subject to the tightest time constraints, when it is probably too late to fix any major architectural problems.

How can agile help here? I would like to see specific stories - and even specific sprints - designed entirely around these more nebulous concepts. This necessitates a “hold the front page” attitude: a willingness to stop development of new functionality, and instead focus on consumability/failover/documentation based on testers’ experience of earlier stories. Perhaps this is a way to implement SVT in agile…

Not much activity on here recently, must be stuff going on somewhere.

Anyway, I’ve been pondering the role of testing in an agile environment more and more recently as its something we need to get right. I’ve already touched on tester driven design in an earlier post.  Now I’d like to look at three basic concepts that may help us move on. The three concepts are:

1. Forget about test phases and replace with ‘user stories’
2. The coding, testing continuum for a user story
3. The roles in an agile team

The first concept is probably the most scary. Most teams are familiar with the Design, Code, Unit Test, Functionally verify, System verify approach to developing software. So the assumption is that in an agile environment you will still need an FV phase followed by an SV phase. This is what I want to challenge.

I’d like us to think of it more simplistically. We write some code, we do some testing. What code and testing we do should be clearly defined by the user stories that are driving the delivery. So, if for example we are delivering a new API, we would have user stories to describe how the API would be used in the single user environment (equivalent to FV type testing) and stories that describe how it might be used in a high load environment (equivalent to SV type testing). So the right type of testing still gets done, we are just driven by the user story deliveries rather than by the traditional test phases.

This then leads nicely into the second concept which I’ll cal the ‘code/test continuum’. When you have user stories as described above you’ll find that some have a large amount of coding and a relatively small amount of testing. (Delivering an implementation of an industry standard API that already has a conformance test suite would be a good example of this). At the same time you will also have user stories that have very little coding, but large amounts of testing required (using the API in a complex environment under heavy load for instance).

So the two extremes may be expressed like this.

CODE………test..

code….TEST…….

But in reality the user story can fit anywhere on the continuum.

This is an important step to realize, because I think traditionally use cases and user stories have been geared towards the functional or code end of the spectrum and we need to ensure that all aspects of a delivery are considered.

Now how do we deliver on this sort of continuum? Well that’s where my third concept comes in. The roles of the agile team. For too long we have had the split in roles between coders and testers. This has to change if we are to be more agile. What we need is skilled software engineers who are capable of coding and testing. If we had a one person project then  it is obvious that that one person would need to perform both roles. so why is it that when our project team s grow  we feel we have to split the roles across different people?

If you take an ideal sized project team (about 10) I’d expect to see a spectrum of people ranging from a small number of pure coders at one end to a small number of pure testers at the other, but with the bulk of the team sitting in the middle as ’software engineers’ who can switch roles depending on the needs of the user story. The team would morph with each delivery making much more effective use of the team’s resources.

I’ll be carrying on this thread as its close to my heart at the moment. Let me know your thoughts.

Big business frequently looks to the sporting world for fresh approaches to improve teamwork, motivation and goal setting. All too often the focus tends to be on the high achievers and their results, whilst overlooking the need to plan, develop and perform more like their sporting heroes.

The modern business trend is to adopt an increasingly flexible or agile business model.

In this article we will examine some of the techniques used by athletes and their coaches and explore how these can be harnessed to support such a business environment.

Read the rest of this entry »

I went up to London this week to attend my first British Computer Society (BCS) event. The Specialist Group in software Testing (SIGIST) hold quarterly conferences open to testers around the UK. Over 250 testers turned up for a day of presentations and workshops. I opted for all the presentations as the theme was Agile testing (my hot topic of the moment), but I hear the workshops were good too. I’d like to attend more of these and hopefully get the opportunity to present.

I picked up a few useful links:

• The QAGuild (Quality Excellence) has some good tooling information

• A collection of Testing blogs called TestingReflections. Well worth a read.

• Another collection of useful Test information and blogs - The UK Tester Forum

I started writing a comment to Ben Bakowski’s last post on “Stressful testing…“, but decided to turn it into a full post. I was thinking that a key difference between testing a “real world” object such as a car, and something like software, is that the limits aren’t always as clear. A car might have five seats, and might use roads where the speed limit is 70mph - these are concrete numbers, which allow you to make the distinction more easily. If I have a piece of software (especially a piece of middleware), then it’s tricky to know the limits. How many records can my database handle? How big can they be? How many users at once?

When testing middleware, I think you’re usually forced to ask yourself: what do I expect someone to do with this? It’s largely about guesswork and experience with existing customers. I think that an interesting idea to explore, is whether it is appropriate or advisable to set artificial limits on a product’s maximum load. Setting artificial limits could provide actual boundaries to the testing that needs to be done. I can think of a few “real world” analogies:

• Putting a “maximum load” sticker on a lorry - ignore at your own risk.
• Putting a fixed number of seats in the car - “we could fit eight seats in here, but we’ve only put in five, so only try and put five people in it”.
• Fitting a speed limiter (i.e. a hard limit).

We could do the same in software: we could write out log messages when a particular level of load is exceeded; we could put in hard limits to the number of records in a database and the number of concurrent users. The key thing when doing this, I think, would be provide reasonable limits which won’t impede your customers, but at least “frame” your testing. For those die-hard customers who absolutely can’t do without that ten millionth database record, you could always offer a special option to remove the limiters, subject to additional testing, or acceptance of the additional risk.

What do you think?  Would this cause as many problems as it fixes?  Or would it enable you to focus on the testing that matters?

In an earlier blog I mentioned that there is a distinction between “load” and “stress” testing which is worth further clarification. This is important - as knowing which regime you’re in helps you understand a test’s value.

Crudely, load and stress testing involve pushing software/hardware at different levels, the latter “harder” than the former. Jon Tilt suggested a good analogy based around testing a passenger car. In a good demonstration of “reuse”, I’ve shamelessly stolen the analogy here…

Read the rest of this entry »

Travelling to the Share conference (share.org) in Orlando I’ve achieved something I have never done before. I started, and finished a book during the flight. And proudly it wasn’t a small book of pictures, it was a book on Software Testing.

Having previously not given it much attention, the book was laying around my office for a while. Recently, I was on the hunt for some information and rather than go straight to the mother of all search engines, I decided to be old-school and check out the books on my shelf.

Flicking through a few I stopped on one that gave me exactly what I was after. In fact, it appeared to me that within its contents were lessons that every Tester should be aware of. It hit me, “Where has this book been all my life?”.

The answer was easy; “Sitting on a shelf”. After all, the book which was written by Glenford J. Myers has been in publish for the last 30 years.

It’s called “The Art of Software Testing”.

Read the rest of this entry »

An age old problem when testing something, is knowing when to stop. We’ve talked a lot about risk in previous entries, but some actual numbers are useful to keep the project managers happy. One measurement that is particularly popular for unit tests is code coverage. This is probably because you don’t care about whether the whole thing works, just the individual units, and you can count them easily enough. For example, the criteria for entry into the traditional test cycle, might include that the developers should have exercised at least 80% of the code. The theory here, is that with detailed knowledge of the code, the developer can exercise all the error paths which are tricky to get to unless you know exactly how. The trouble is, that with deadlines looming, measurements like code coverage can easily encourage people to test the easiest 80% - in this situation, you have a developer who has tested every “getter” and “setter” method that they can find, but haven’t ventured into the bit of code that only gets run when it’s a full moon.

Of course, it’s good to have some unit testing done before functional testing begins, but the purpose should surely be just to check that the whole thing isn’t going to fall apart as soon as somebody looks at it funny. Perhaps a better place to measure code coverage is after the black box tests have been run. If the developers could do some basic tests on the product, to at least prove that the product works sometimes, that would probably be good enough as a test entry measurement. Then, we could examine the amount of code that is covered by all the black box tests, which are trying to hit all the problems that customers might hit. Now, armed with the information about which bits of the code we know aren’t tested, and which we know are difficult to hit, we can start to use white box tests more effectively.

Of course, you still need to know when that first wave of “bring up” tests are done, so that the more formal testing can begin. This might be hard to quantify, so this approach does imply that you might not get any concrete test measurements until later in the cycle - of course, if the current measurement of unit test code coverage is actually providing false confidence, then maybe this isn’t a bad thing.

I’d like to thank one of our senior developers for inspiring this post. Dave Vines recently sent out an excellent, thought provoking email on how we should be moving towards a tester driven development model. I’ve slightly reworded his email:
“One thought that I’ve recently had was that perhaps we need “Tester Driven Development” rather than “Test Driven Development”. Let me try to explain.

We are currently very much a “Developer Driven Development” organization:
We gather new requirements and (typically) farm them out to the development community to investigate
Once a work item is identified, it is typically a developer that

• Writes the Design document
• Calls the meeting to develop the use cases
• Identifies the interesting scenarios
• Gathers the development resource to write the code
• Decides which iteration in which the code will be written
• Hopes that test will have the resource to test all the use cases and “interesting edge cases” that have been identified and for which code has been written (or not written)

In effect the developer owns the work item

In a “Tester Driven Development” organisation I would see us:
Gathering new requirements and (typically) farm them out to the test community to investigate
Once a work item is identified, it is typically a tester that:

• Writes the design document
• Calls the meeting to develop the use cases
• Identifies the interesting scenarios
• Gathers the test resource to test those scenarios and use cases
• Decides which iteration in which the code will be tested
• Indicates to development what use cases and code will be needed and when

In effect the tester owns the work item

This would be a massive culture shift for a department, but one that makes it clear that the desired deliverable is actually requirements that have been tested, not requirements that have been coded and may have been tested if test had enough resource. “

Read the rest of this entry »

Few software engineers would argue against the importance of automation, but the code that tests code is often hidden away and shies from the limelight. Automation tools receive extensive and repetitive use, and as a result, they are generally very well tested. Without doubt this is an innovative space, the things we test are sophisticated which demands great sophistication and creativity from a test harness.

Test automation is certainly not unique to the software industry. The things we use every day have all been tested in some way, many in an automated fashion. I always become engrossed with the Dyson testing advert when I catch it on TV. They have put their test automation tooling on centre stage and it is fascinating to watch in action.

http://www.dyson.co.uk/testing/