Unit testing; how far do you push the envelope?
December 4th, 2007 by Oscar HuseyinOver the years, l’ve read lots of commentary, white papers, best practice papers, books on the topic of TDD. I’ve heard the rant of many TDD evangelists who preach about how total code coverage brings you closer to code quality perfection and how you’ve failed when you’ve not been able to achieve these goals. Sure, this is an extreme example of evangelical preaching, where in actual fact, most of these individuals commonly drum down their hard line views of testing by using words like “pragmatism” and statements like “do what works best”. But, why do l feel as if I’ve failed if l have not got 100% code coverage? It’s because l, to some degree, shared some of the religious views about testing.
I’m now at a point where I’m beginning to rethink some of my beliefs about testing after many years in the trenches. So, l’m at the crossroads settling on a methodology that, l feel, works the best. What level of unit testing is really required to meet the business needs?
I’ll start by analysing two “special interest” projects to see what the outcomes they delivered based on the business expectations.
Firstly, let me talk about a project that was one extreme; no mandated position on unit testing. The project was highly successful, where the business expectations were met and exceeded. Donning my evangelist hat, I’d say the project outcomes were a fluke and it was a miracle that we were able to make any changes to the application without having a negative impact on functionality. Looking back, the project was definitely not a fluke; we made lots of changes to the application without any regressive impact. We knew our issues and had the right processes in place to gate-check the application functionality pre-release. For example, a week before each release, every developer had an area of expertise in the application which they would spend approximately a week testing the functional area and making any spot fixes as need be. We were not very clever about our testing methodology, but we delivered on time, on budget and exceeded customer expectations.
Now, let me tell a story of another, very different project, one that’s in stark contrast to the first one. This application had literally 98% code coverage. Unit tests, integration tests, front end screen tests, water tight code reviews, continuous integration, nightly deploys, every agile practice and quality assurance process under the sun. Did the code meet the business expectations? Well, yes; but it was expensive. It took twice as long to develop an application feature, and we mandated near perfect code coverage. Was this approach more successful than my first example? Not really. Sure, we had more confidence in making changes to the code base and having an “immediate view” of regression impact of the change. But the business paid a price for all of that. A very heavy price. One would think, given the money it cost for development to test the application, that the number of defects would be significantly reduced; but they weren’t. We had lots of functional and non-functional defects detected by testing which was effectively misinterpretation of business requirements or some gap in the business logic.
Which, from a business perspective, was more successful? Both. The corollary is that a heavily tested application cost lots of money and takes longer to build. This l have seen first hand. So, time to answer the titan question from my own experiences.
As a developer, you need to test the components that you write; theres no arguing that. Otherwise, how else can you prove the functionality of your components? Bu, just how far should we push the envelope?
My view is simple. We all need to be pragmatic about how we approach our unit testing. We should always stop and ask ourselves “are we going to far with our unit testing?”. As a developer, we are faced with this question constantly. We should always do the most to prove our components are functionally correct, but also write the least amount of unit test code to ensure our testing solution remains simple yet effective. After all, a good developer is a lazy one.
December 7th, 2007 at 1:23 pm
The best thing I like about TDD is that it really does help you to do it once and do it right. As you say, a good developer is one who is experienced enough to know what to test and lazy enough to test that only. Developers who are aware of their capabilities and limitations are generally confident in the code that they write and know that when they’re in doubt they must test. If you work on one thing at a time you can unit test as you go. Code a little, test a little, code a little, test a little..and so on until the job is done. If you write and maintain your tests as you go you can be confident that your code will work when you’re done. By keeping it all simple you can never go too far with your testing.
December 7th, 2007 at 7:08 pm
It’s totally developer perspective that how he wants to unit test. Standard unit tests can be defined by the Team Lead but after that it’s like self review.
December 8th, 2007 at 6:12 am
Unit testing; how far do you push the envelope?…
[...]Developers are paid to write working code and are expected to be productive. Heavily tested applications can take a long time to build and can cost businesses lots of money. To be successfully productive, developers must gauge what to test and w…
December 8th, 2007 at 6:49 am
hi there,
I’ve thought about why it “testing adequately” is so much more expensive ? The answer is that ,
1) developers don’t have adequate testing experience
2) developers don’t have adequate tools
a) languages
b) enough hardware
if we start using the right testing frameworks/languages/hardware with the right training, then we would get better value for money/time invested in unit/functional/integration testing; Especially if QA team can reuse some of the development team artifacts OR atleast some of the test use cases.
BR,
~A
December 8th, 2007 at 9:40 pm
I don’t believe that tools are the answer. Most of the time I develop code I - think! So I like TDD because it changed my thinking. It forces you to think about decoupling, and it makes design decisions about “how do I want to use the class” explicit.
It enables me to refactor my code without fear. This helps to reduce “big refactoring” time, which is always more expensive than you’d guess.
Test coverage in terms of “functional requirements” can’t come from your
unit tests, but only from higher level tests which you have to write /on top/.
Unit tests are not enough, even if they yield 100% test coverage.
Cheers,
Manuel
December 9th, 2007 at 7:48 am
[...] Unit testing; how far do you push the envelope? (odecee blogs) is a interesting read on the issue of how much unit test code is enough. [...]
December 9th, 2007 at 9:36 am
How far do you test? My rules are thumb are:
1. Don’t test simple beans.
2. Don’t test dependency injection.
3. Mock your collaborators in your tests.
4. Test your core logic but be pragmatic about this.
5. Don’t test for developer stupidity (unless writing a 3rd party API)
6. Trust your team. Review their tests, don’t tell them what to write.
7. A bug is fixed only when you have a test that proves it is fixed.
Every developer has their own viewpoint and
December 19th, 2007 at 8:49 am
I have lived through exactly the same project (your second example) - although it cost more than 3 times as much, and took a lot longer to build than expected.
This project was “super-agile” - doing TDD with 100% unit test coverage… the code base ended up being 90% unit tests and 10% actual code. I think there was even unit tests for the unit tests!
I think unit testing and testable code is something that is required, but this project showed me that if you put a bunch of agile evangelist nutters in a room together you’re not gonna get something good at the end of it….
December 19th, 2007 at 9:56 am
Interesting old-new thought. I come from a QA development side of Software Engineering therefore I am a TDD evangelist, however, I believe that one should not test everything but test what might break. This is sometimes difficult to determine. However, I have found that having in place an automated infrastructure to cover special cases such as entitlement usage, producer-consumer, and data serialization will save you from some headaches.
I would not recommend any one to go crazy testing everything because you could produce a tremendous number of test cases and never find a bug. Use your strength in a smart way because building software is like running a marathon.
The other super problem that I have seen is to have a bunch of little test apps or testing in a main function. That is bad: stick the tests into JUnit and commit them so every one can use, and re-use these tests.
December 19th, 2007 at 11:41 am
The idea that somehow 100% coverage is somehow equal to a “full test” of the program is probably the most ridiculous myth out there. Consider a trivial example like
public String getFirstFile(String path) {
return new File(path).list()[0];
}
You run getFirstFile(”c:\\windows”) and you get 100% coverage. Does this mean that you have tested the full spectrum of software behaviour? Certainly not - if the path does not exist, you’ll have a NullPointerException that went undetected because of the 100% coverage meme.
100% coverage is just ONE of the techniques ensuring program stability, and it should be easily achievable and be considered simply a baseline. There are many other powerful testing techniques (Equivalence Partitioning, Boundary Value Analysis, etc.)
December 19th, 2007 at 4:07 pm
Oscar, thanks for sharing your experiences. I’ve written also a blog about very similar topic; some agile (Scrum) teams are putting coverage commitment to their Definition of Done and that leads to number of problems. Check it out at http://huitale.blogspot.com/2007/11/common-scrum-pitfalls-killing-your.html
TDD is a practice that requires team to have lots of experience and even more professional discipline. That’s why I think it is good to challenge the TDD practice (Abrahamsson’s & Sinisalo recent studies) and teams capability to follow the practice (Bas Vodde & Lasse Koskela @ IEEE SW “by definition TDD leads to horrible design if not refactored continuously”).
December 19th, 2007 at 6:38 pm
What about the idea that the tests provide some kind of functional documentation? I know it’s not ideal and does not replace ‘real’ documents but the fact that someone has tested his/her code by writing a test also defines the expected functionality of that module. For developers joining or inheriting the project later this can be a great advantage. How often does formal documentation fall behind the current code? Unit tests can help to compensate for this. (That’s my excuse for not maintaining the docu anyway )
December 19th, 2007 at 11:18 pm
Here’s a question. After the code was put in use, how difficult was it to maintain/change each of the applications?
December 20th, 2007 at 1:32 am
How large were the projects in term of project team size and duration? I ask because in my experience, shorter projects can get away with less testing. It’s easier to wrap your brain around the whole thing, you can more reliably predict how code changes will ripple through the system, and there hasn’t been enough time to forget what you’ve done.
In larger projects, 6+ months or with larger teams and large codebases, I found the unit test suites to be incredibly helpful. Furthermore, I think the problems that project suffer from without good testing grow exponentially with size. Ten months into a project with 20+ developers, I’ve seen vicious cycles of code breakage because there are too many moving parts and no way to predict what breaks when something changes. You rarely see small projects and small project teams encounter this.
December 20th, 2007 at 4:47 am
[...] Oscar Huseyin, via JavaLobby, has an interesting post on how much unit testing is good: [...]
December 20th, 2007 at 9:08 am
[...] Unit testing; how far do you push the envelope?- Good advice- “write the least amount of unit test code to ensure” things work well. [...]
December 26th, 2007 at 9:20 pm
Unit testing is great if you can say anything of business value about the unit. Most of TDD is based on xUnit, and the unit of any given test is a procedure. So if the procedures have business value, then you’re in good shape. I think TDD makes sense at the outermost layers of objects in a system that is imperatively driven, where their object-ness gives way to the business functionality of the system. It also makes sense in procedural systems. TDD dates back to the early 1960s, and it worked well on FORTRAN programs, particularly those built in a bottom-up fashion, where the business requirements could be broken down into low-level functions. That doesn’t make sense in many of today’s interactive systems, but it still makes sense in systems where algorithms (instead of objects) are the primary organizing principle
However, this implies that TDD doesn’t make sense for a system where objects are the primary organizing principle. I don’t know how to test an object. Objects are about structure; what needs to be tested is the degree to which they map the user’s cognitive model of the domain. Such are the origins of OO, and such are the hallmarks both of a maintainable OO design and of a design with good usability. Such reasoning is the very foundation of MVC. Testing the functionality of object member functions causes one to focus on the functions and lose sight of the objects, and—hey, you can write FORTRAN in any language. We’ve seen this many times in our own projects. And now work by Pekka Abrahamsson and Maria Siniaalto has uncovered the same results in some research projects.
So, yes: the cost of unit testing is high, not only in the short term, but in the time scale that corresponds to architectural costs and benefits. Furthermore, I’ve yet to see a good study that quantifies the *benefits* of unit testing in an OO context—and certainly not relative to other techniques like pair programming, good old-fashioned code inspections, clean room, and others. In the past five years I have yet to run into a single person who is doing it for other than one of two reasons: that it makes them feel good about themself, or that they are a lemming.
January 2nd, 2008 at 12:14 am
I think the most revealing part of you article was
“We had lots of functional and non-functional defects detected by testing which was effectively misinterpretation of business requirements or some gap in the business logic.”
I am completing a project where I don’t have 100% coverage in unit testing, however the critical areas did. It was easy to write Unit tests for those areas and actually saved time and money. However, most of the “defects” in the project were caused by a misinterpretation of business requirements. And this was caused because once an initial (one day) gathering of requirements were made, we were not allowed to talk to the users (except through the project leader) and the users didn’t see the final result until almost time to turn the project over.
It would be nice to have a better methodology for “unit” tests on requirements. This would require a more “hands on” approach from the users and more user developer interaction. It has to include a process for not creating scope creep (I’ve been involved in a project where there was a daily scope creep due to too much interaction.)
Unit testing is important. Properly done, it prevents changes in one area of our project from affecting another area. But possibly “embedding” a user in the development process is also necessary. To ensure that there is no “misinterpretation” of requirements. And totally isolating the developer from the users is a red-flag!
January 3rd, 2008 at 8:12 pm
Good code is facilitated by TDD but high test coverage does not necessarily mean good code. This is why I don’t believe in setting minimum coverage percentages on my teams. I don’t want my developers thinking that their aim for the day is to get 100% coverage, I want them to think that the aim is to write good code. Writing tests is easy. Writing the right tests is hard. It requires lots of experience and a deep understanding of the cost and benefit of each test — it’s fine line between pleasure and pain!
January 17th, 2008 at 10:15 am
I’ve just read the only accessible papers I’ve been able to find by Siniaalto (the ones from the Itea project) and they don’t appear to say much at all. Her summary of existing TDD research is, if anything, slightly positive about TDD — although everyone admits that the numbers are weak. Her comparative case study is based on student programmers who said they wished they’d been trained properly and wanted to learn more at the end of the project. Hardly a rigourous indictment of TDD. Is there more data in the pay-per-view papers?
In the meantime, Keith Braithwaite has been doing some interesting measurement on established open-source projects and finding interesting, but as yet unexplained, differences between those written Test-First and those not. It’s not clear what’s happening, but the early conventional quality numbers suggest that the TDD projects have better structure.
January 17th, 2008 at 10:25 am
Can I ask about the TDD project? Was it the team’s first “full-on” TDD project? Rather like your posting about Spring, we all tend to overshoot on the first one. I’ve seen other teams hobble themselves by going just a bit too far with the quality stuff–or perhaps a better explanation is, by not quite digging deep enough to find the deeper motivations for mandating tools and processes.
Of course, as Alistair Cockburn likes to point out, a good team of people will always ship. Years ago I worked on a rock-solid project where I, for example, ported the whole thing from single- to double- byte without automated tests. On the other hand, we lost days recovering from individuals’ occasional late night hackery. I wouldn’t do that now.
January 27th, 2008 at 7:36 pm
Hey everyone,
Ive just read a very interesting post outlining and describing a statistical study into TDD. A must read for all TDD proponents!
http://scruffylookingcatherder.com/archive/2008/01/22/tdd-proven-effective-or-is-it.aspx
January 27th, 2008 at 7:39 pm
Steve Freeman,
The project was definitely not the first TDD project for the team. I was working with a company (without mentioning any names) who are very “passionate” about TDD so the process was certain to be TDD oriented.