Integration Tests are a Scam

On my first day at the Agile 2009 conference, and after my presentation there on “The Amazing Team Race”, I attended a session by J.B. Rainsberger titled, “Integration Tests are a Scam.” He wasn’t suggesting that integration tests aren’t needed, but rather the way they handled today by the majority of organizations is a lie; a scam. The title, an assertion, was a bit shocking for me but I attended mainly because the teams I currently manage face several challenges with regards to testing. What follows is what I learned from Rainsberger.

One main problem with integration tests have to do with excessive test setups. This causes:

• Messy test structures.
• Assertions to be repeated throughout the structure.
• Programmers/testers to check all the different places to determine where the test precisely failed and why.

Another problem is that integration tests take a long time to execute. Is there a way to address this? Well, one problem Rainsberger asked is to determine how many tests are actually needed to consider your testing truly complete? Assume we have a large-scale application to test, and after considering all the combinations for complete test coverage we come up with 4 million tests. That’s 4 million tests we will never ever write. Therefore if we won’t ever write that many tests, what tests should we even bother to write? Some suggest writing a small percentage of the total possible tests that are geared to test 80-90% of the application’s functionality. Rainsberger throws out 2,000 tests as the number of tests that a team may actually consider writing for this large application. 2,000 tests out of 4 million only cover less than 1 percent of the total possible number of tests. Even by selecting the “right” 2,000 “focused” tests to write, your chances for good coverage run somewhere in the range of 20-80%. And how do you know you’ve covered 20-80% of the application’s functionality in your integration tests? You DON’T!

Here’s where J.B. introduces the concept of “Basic Correctness.” What is “Basic Correctness” of an application? His definition is “…given the myth of perfect technology, do we compute the right answer?” Integration tests should NOT be needed to prove basic correctness of an application. His solution is, “…write only focused object tests for programming tests… this deals with 98% of the problems, leaving you to use the majority of efforts to address the 2% of tests that fail that aren’t address by your focused object tests.”

Steps to writing good focused object tests are:

• Don’t test the platform. Trust the equipment and libraries that are being used.
• When writing a single object to test with four other collaborating objects/points, use four interfaces for those four other points. Interfaces are not the actual collaborating object you are testing, but merely something that behaves like the actual collaborating object. J.B. also notes to only use interfaces for services.
• Leverage the interfaces so you don’t need to actually test the other objects/points.
• Test the single object to speak to itself, aka State Tests.
• Create your focused Collaboration and Contract Tests.

Above is a diagram for what J.B. describes in setting up focused object tests.

What are Collaboration and Contract tests? Well to start, Collaboration tests are what J.B. refers to what others traditionally call integration tests. To have your collaboration tests be focused, you only need to ask two questions:

• Do I ask the collaborators the right questions?
• Can I handle the collaborators’ responses?

For example, perhaps your function needs to determine how many upcoming customer birthdays are there. Your focused test needs to ask of its collaboration point(s) to find the number of customers with a birthday in the next seven days. Your focused tests needs to handle the following responses: zero, one, many, or lots (for capacity and performance reasons perhaps).

When writing your focused collaboration tests, it’s important to handle the responses from the interface and not the actual object/point. Don’t do the backend work for the test. Leverage the interface.

Contract tests are a test for the interface you are leveraging. It’s also a systematic way for setting up tests reliably. They complete the other side of the test and in essence fulfill a contract. The two questions to ask to create a focused Contract test are:

• Can the interface accept the question being asked of it?
• Can the interface supply the responses expected?

In the end you have focused object tests that are succinct and complete in the verification of both what is being asked of its collaborators as well as ensuring that your collaborators can complete their side of the contract by supplying what is needed.

As shown in the diagram above, Rainsberger tells us that the number of tests needed is directly correlated to the number collaboration and contract tests per interface for each related collaborator object to the focused test object. This also eliminates the worry of the numerous combinations of code paths to tests. Therefore when you find a test failure, you only need to check in two places. So when referring back to the example J.B. gave us, as I mentioned in the beginning of this post, about an application with the need to have 4 million tests in order to have complete test coverage:

• The need to create 4 million tests for complete test coverage has been brought down to 10,000.
• Performance time in executing the tests is now shorter.
• Actual effective test coverage can be achieved.
• Pushes engineering towards a good modular design.

Your integration tests are sound and complete when you have:

• State tests
• Collaborator tests
• Contract tests

Now J.B. admits that there may an issue in automating contract tests, but he suggests that there may be some workarounds.

Finally the benefits, which he says he has not been proven but that some PhD potentially could, are:

• When integration tests fail, engineering will only need to check in two places for where and why the failure occurred.
• Integration tests only need to be written into Collaboration tests and Contract tests.
• There is no need to write integration tests during design.

J.B. Rainsberger is heavily involved with the Agile community and has spoken at many conferences. To learn more or follow J.B., be sure to check out his blog at: jbrains.ca.

To see a sample of what J.B. means by Contract tests click here.

Agile 2009

This year's Agile conference in Chicago certainly did not disappoint. Besides presenting at the conference with my former colleague Belkis McCall-Vasquez on "The Amazing Team Race", there was plenty to learn and plenty of exciting people to meet.

This week, before Labor Day, I plan to devote my posts to highlighting some of the interesting lessons learned during my week at the Agile 2009 conference. Some of these highlights include:

• J.B. Rainsberger's thoughts on how most organizations' integration tests today are a scam and how they can be more focused and effective.
• Pete Behren's recommendations in running effective and successful Scrum meetings.
• David Hussman's savvy approach in coaching and producing value for organizations adopting Agile.
• Renzo Borgatti's presentation on leveraging the Pomodoro technique to get your teams focused on completing their tasks on a daily basis.
• Luke Hohmann's approach to prioritizing your backlog for profit.
• Scott Ambler's myth-busting approach to statistics he's found with organizations using Agile.

Finally, the entire conference was honestly a great learning experience and even a lot of fun. It included Muzik Masti music every night, engaging networking events (especially those put together by ThoughtWorks), and a fun night on a Lake Michigan cruise around Chicago with fireworks hosted by VersionOne. To the right is a picture of Jeff Patton, Belkis McCall-Vasquez, myself, and Pete Behrens in front of the Mystic Odyssey for our cruise that evening.

Ri - The Japanese Word for Invent or Blend at the Agile 2009 Conference

An interesting talk given by Alistair Cockburn during his keynote speech at the Agile 2009 conference today. His speech was titled, "I Come to Bury Agile, Not Praise It." Don't be mistaken. He's not suggesting that Agile is dead, but rather that we all need to seek to effective means to apply Agile principles to our circumstances. One idea from his presentation I'd like to highlight here is his thoughts on the Japanese words of:

• Shu - meaning to learn, a technique that can be generally applied.
• Ha - meaning to collect, techniques to be applied.
• Ri - meaning to invent or blend, techniques to be applied.

The Japanese characters for "Shu, Ha, Ri".

All too often, many of us (especially novices) look to learn a specific technique to apply to our own work and circumstances. Alistair suggests that the "Shu", or the learning of a technique is just the beginning. When you learn one technique and apply it to your project work, you will find that it may work but not necessarily optimally. When you seek to refine your technique, some tend to "Ha," or collect a string of techniques to see which can be applied to his/her projects and work environment. Kind of like having a toolbox of tools to use that could be applied to each unique circumstance. Alistair challenges us as professionals to leverage "Ri" and invent/blend techniques so that they are customized and are compatible for the unique circumstances your work environment needs. It is only when we experiment and blend our learned/collected techniques that real innovative value is achieved. You may not at first understand what it is you are actually doing, but clarity will be attained with your results.

"Ri" will be my mantra during my attendance at this week's Agile 2009 conference.

Innovation Strategies: Facebook's Acquisition of FriendFeed

The Wall Street Journal (WSJ), in collaboration with MIT's Sloan Business School, came out with a Journal Report on Innovation this past Monday. It's a must read for most business professionals, especially for those who work in areas with a heavy focus on R&D and technology. Reflecting on my 12-year history with software technologies, especially on my enlightening time at McKinsey & Company, I learned the importance of innovation. Now that I'm currently working on some green field projects that are providing new products and services within the financial industry, I also see the importance of the strategies employed to seek innovation and stay competitive while maintaining low costs.

In leveraging certain strategies, some business executives believe they gain innovation, a competitive edge, and cost benefits through acquisitions of other organizations. In reality, after the attrition of some of the workforce during the re-organization that always eventually ensues, the business only gains innovative products and short-term cost benefits. Products are not the source of an organization's innovation. It is the talent pool of laborers within the organization, whether in R&D, engineering departments or elsewhere. I see this struggle when financial companies, seeking to provide such things as innovative high-frequency trading services, under appreciate the value of the engineering talent pool and focuses solely on the headcount costs. Losing talent, in many cases, also means losing domain knowledge and the innovation that comes with it.

In my first post regarding innovation, I'd like to point to a recent event in the technology world where a specific strategy to gain innovation was used, the acquisition of FriendFeed by Facebook (see TechCrunch's take on the acquisition here). Now FriendFeed had 989,000 unique users in June. Compare that to Facebook's 340 million, all according to comScore, and you can see that the acquisition wasn't done for market share. There was also a clear overlap in features and functions in what both Facebook and FriendFeed offer. So what was Facebook's angle in doing the acquisition? Talent acquisition. FriendFeed's 12 employees, all but one is an engineer, clearly demonstrated their ability to quickly innovate and deploy new products. Products that raised the bar on what Facebook could currently offer with it's comparable features. This talent pool was the asset that Facebook sought, especially in their quest to competitively battle Twitter. (Note: Facebook failed to acquire Twitter last year.)

Although the future of the FriendFeed application remains uncertain, the belief is that the talent pool from FriendFeed will integrate the existing features onto the current Facebook application and further their innovative design mind-set within the engineering organization at Facebook. In their WSJ Journal Report article on innovation, "Finding an Innovation Strategy That Works", Frank Rothaermel and Andrew Hess state, "...we found that the most effective way to achieve continuous innovation over the long term is to hire and cultivate talented people."

To learn more about Facebook's need to absorb FriendFeed's innovative features, as written by CNet, click here.

Gain Market Share & Early Adopters of Technology by Being Free

Students at universities all across the country have been complaining that their inboxes are too small for the tsunami of emails that they receive on a daily basis. As a result, many universities have been outsourcing email services to companies such as Google and Microsoft. Most entrepreneurs and businesses would see a profitable business opportunity to charge at least small fee for such services to universities, just as several already do for companies. The interesting thing is that Google and Microsoft charge nothing to the universities for email services.

Universities benefit in cost savings by avoiding costly equipment upgrades/maintenance and number of employees managing information technology services. Also students' satisfaction with a university's outsourced technology services, in this case email, dramatically increased.

So what's in it for Google and Microsoft? College students are generally early adopters of new technology. The idea is to gain the participation and feedback of students for other web services, beyond email, being developed and deployed by Google and Microsoft. The hope is to gain early acceptance and adoption for those products/services and have them become mainstream. This approach doesn't only apply to companies following a Web 2.0 model. Chris Anderson talks about such strategies leveraged by cutting edge companies in his book, "Free: The Future of a Radical Price."

So...got a new idea? Desire early and free feedback to perfect your product or service? Want to quickly gain market share? Then consider rushing out to offer it for free to the students of your nearest university before your competitors do. Come to think of it, didn't Facebook start off as a college social networking site?

Want to know more? Click here to read the Time news article on free web services for students at universities.

Evolution of: Ctrl + Alt + Del

Here's retired Dave Bradley of IBM, the inventor of the "Ctrl + Alt + Del" shortcut and how it evolved to become so popular. Interesting reaction by Bill Gates...

Starbucks Goes 'Lean'!

In an article on the front page of today's Wall Street Journal, Starbucks is leveraging a new discipline and practice to their culture, 'Lean.' After making some comparisons on the ability of their fast food chain competitors (McDonald's and Dunkin Donuts) to service more customers in shorter time, it seems that now Starbucks sees the value in the Toyota Production System (TPS) where lean process and fast delivery, without compromising on quality, is paramount.

What does this prove? That lean processes, such as Agile or TPS, are not strictly defined methodologies that may only prove to work in certain environments, but that they are mere frameworks that can be tailored to work for your organization's needs regardless of the industry. In the end the idea is to:

1. Streamline the waste and needless bureaucracy.
2. Free and empower your organization's members to focus on delivering the highest business value.
3. Enable your teams to become high performing.

Any thoughts on how lean practices could benefit other organizations in industries outside of automotive, technology, or food services?

Check out the WSJ article on Starbucks leveraging 'Lean' practices here.