Programmers
We eliminate build and configuration hassles.
Here’s an ideal to strive for. Imagine you’ve just hired a new programmer. On the programmer’s first day, you walk him over to the shiny new computer you just added to your open workspace.
“We’ve found that keeping everything in version control and having a really great automated build makes us a lot faster,” you say. “Here, I’ll show you. This computer is new, so it doesn’t have any of our stuff on it yet.”
You sit down together. “OK, go ahead and check out the source tree.” You walk him through the process and the source tree starts downloading. “This will take a while because we have all our build tools and libraries in version control, too. Don’t worry—like any good version control system, it brings down changes, so it’s only slow the first time. We keep tools and libraries in version control because it allows us to update them easily. Come on, let me show you around the office while it downloads.”
After giving him the tour, you come back. “Good, it’s finished,”
you say. “Now watch this—this is my favorite part. Go to the root of
the source tree and type build.”
The new programmer complies, then watches as build information flies by. “It’s not just building the source code,” you explain. “We have a complex application that requires a web server, multiple web services, and several databases. In the past, when we hired a new programmer, he would spend his first couple of weeks just configuring his workstation. Test environments were even worse. We used to idle the whole team for days while we wrestled with problems in the test environment. Even when the environment worked, we all had to share one environment and we couldn’t run tests at the same time.
“All that has changed. We’ve automated all of our setup. Anybody can build and run all the tests on their own machine, any time they want. I could even disconnect from the network right now and it would keep building. The build script is doing everything: it’s configuring a local web server, initializing a local database... everything.
“Ah! It’s almost done. It’s built the app and configured the services. Now it’s running the tests. This part used to be really slow, but we’ve made it much faster lately by improving our unit tests so we could get rid of our end-to-end tests.”
Suddenly, everything stops. The cursor blinks quietly. At the
bottom of the console is a message: BUILD
SUCCESSFUL.
“That’s it,” you say proudly. “Everything works. I have so much confidence in our build and tests that I could take this and install it on our production servers today. In fact, we could do that right now, if we wanted to, just by giving our build a different command.
“You’re going to enjoy working here.” You give the new programmer a friendly smile. “It used to be hell getting even the simplest things done. Now, it’s like butter. Everything just works.”
What if you could build and test your entire product—or create a deployment package—at any time, just by pushing a button? How much easier would that make your life?
Producing a build is often a frustrating and lengthy experience. This frustration can spill over to the rest of your work. “Can we release the software?” “With a few days of work.” “Does the software work?” “My piece does, but I can’t build everything.” “Is the demo ready?” “We ran into a problem with the build—tell everyone to come back in an hour.”
Automating your build is one of the easiest ways to improve morale and increase productivity.
Sadly, build automation is easy to overlook in the rush to finish features. If you don’t have an automated build, start working on one now. It’s one of the easiest things you can do to make your life better.
There are plenty of useful build tools available, depending on your platform and choice of language. If you’re using Java, take a look at Ant. In .NET, NAnt and MSBuild are popular. Make is the old standby for C and C++. Perl, Python, and Ruby each have their preferred build tools as well.
Your build should be comprehensive but not complex. In addition to compiling your source code and running tests, it should configure registry settings, initialize database schemas, set up web servers, launch processes—everything you need to build and test your software from scratch without human intervention. Once you get the basics working, add the ability to create a production release, either by creating an install file or actually deploying to servers.
Construct your build so that it provides a single,
unambiguous result: SUCCESS or
FAILURE. You will run the build
dozens of times per day. Manual checks are slow, error-prone,
and—after the umpteenth time—seriously annoying.
You should be able to build even when disconnected from the network.
A key component of a successful automated build is the local build. A local build will allow you to build and test at any time without worrying about other people’s activities. You’ll do this every few minutes in XP, so independence is important. It will also make your builds run faster.
Be cautious of IDEs and other tools that promise to manage your build automatically. Their capability often begins and ends with compiling source code. Instead, take control of your build script. Take the time to understand exactly how and why it works and when to change it. Rather than starting with a pre-made template, you might be better off creating a completely new script. You’ll learn more, and you’ll be able to eliminate the complexity a generic script adds.
The details are up to you. In my build scripts, I prefer to have all autogenerated content go into a single directory called build/. The output of each major step (such as compiling source code, running tests, collating files into a release, or building a release package) goes into a separate directory under build/. This structure allows me to inspect the results of the build process and—if anything goes wrong—wipe the slate clean and start over.
At the start of the project, in the very first iteration, set up a bare-bones build system. The goal of this first iteration is to produce the simplest possible product that exercises your entire system. That includes delivering a working—if minimal—product to stakeholders.
Use the build script to configure the integration machine. Don’t configure it manually.
Because the product is so small and simple at this stage, creating a high-quality automated build is easy. Don’t try to cover all the possible build scenarios you need in the future. Just make sure you can build, test, and deploy this one simple product—even if it does little more than “Hello, world!” At this stage, deployment might be as simple as creating a .zip file.
Once you have the seed of your build script, it’s easy to improve. Every iteration, as you add features that require more out of your build, improve your script. Use your build script every time you integrate. To make sure it stays up-to-date, never configure the integration machine manually. If you find that something needs configuration, modify the build script to configure it for you.
If you want to add a build script to an existing system, I have good news and bad news. The good news is that creating a comprehensive build script is one of the easiest ways to improve your life. The bad news is that you probably have a bunch of technical debt to pay off, so it won’t happen overnight.
As with any agile plan, the best approach is to work in small stages that provide value as soon as possible. If you have a particularly complex system with lots of components, work on one component at a time, starting with the one that’s most error-prone or frustrating to build manually.
Once you’ve picked the right component to automate, start by getting it to compile. That’s usually an easy step, and it allows you to make progress right away.
Next, add the ability to run unit tests and make sure they pass. You probably compile and run unit tests in your IDE already, so this may not seem like a big improvement. Stick with it; making your build script able to prove itself is an important step. You won’t have to check the results manually anymore.
Your next step depends on what’s causing you the most grief. What is the most annoying thing about your current build process? What configuration issue springs up to waste a few hours every week? Automate that. Repeat with the next-biggest annoyance until you have automated everything. Once you’ve finished this, congratulations! You’ve eliminated all your build annoyances. You’re ahead of most teams: you have a good build script.
Now it’s time to make a great build script. Take a look at how you deploy. Do you create a release package such as an installer, or do you deploy directly to the production servers? Either way, start automating the biggest annoyances in your deployment process, one at a time. As before, repeat with the next-biggest annoyance until you run out of nits to pick.
This won’t happen overnight, but try to make progress every week. If you can solve one annoyance every week, no matter how small, you’ll see noticeable improvement within a month. As you work on other things, try not to add new debt. Include all new work in the build script from the beginning.
A great build script puts your team way ahead of most software teams. After you get over the rush of being able to build the whole system at any time you want, you’ll probably notice something new: the build is slow.
With continuous integration, you integrate every few hours. Each integration involves two builds: one on your machine and one on the integration machine. You need to wait for both builds to finish before continuing because you can never let the build break in XP. If the build breaks, you have to roll back your changes.
A 10-minute build leads to a 20-minute integration cycle. That’s a pretty long delay. I prefer a 10- or 15-minute integration cycle. That’s about the amount of time it takes to stretch my legs, get some coffee, and talk over our work with my pairing partner.
The easiest way to keep the build under 5 minutes (with a 10-minute maximum) is to keep the build times down from the beginning. One team I worked with started to look for ways to speed up the build whenever it exceeded 100 seconds.
Many new XP teams make the mistake of letting their build get too long. If you’re in that boat, don’t worry. You can fix long build times in the same agile way you fix all technical debt: piece by piece, focusing on making useful progress at each step.
Slow tests are the most common cause of slow builds.
For most teams, their tests are the source of a slow build. Usually it’s because their tests aren’t focused enough. Look for common problems: are you writing end-to-end tests when you should be writing unit tests and integration tests? Do your unit tests talk to a database, network, or file system?
You should be able to run about 100 unit tests per second. Unit tests should comprise the majority of your tests. A fraction (less than 10 percent) should be integration tests, which checks that two components synchronize properly. When the rest of your tests provide good coverage, only a handful—if any—need to be end-to-end tests. See Speed Matters in Chapter 9 for more information.
Although tests are the most common cause of slow builds, if compilation speed becomes a problem, consider optimizing code layout or using a compilation cache or incremental compilation. You could also use a distributed compilation system or take the best machine available for use as the build master. Don’t forget to take advantage of the dependency evaluation features of your build tool: you don’t need to rebuild things that haven’t changed.
In the worst-case scenario, you may need to split your build into a “fast” build that you run frequently and a “slow” build that an integration server runs when you check in (see Continuous Integration,” later in this chapter). Be careful—this approach leads to more build failures than a single, fast build does. Keep working on making your build faster.
Who’s responsible for maintaining the build script?
All the programmers are responsible for maintaining the script. As the codebase evolves, the build script should evolve with it.
At first, one person will probably be more knowledgeable about the script than others. When you need to update the script, pair with this person and learn all you can.
The build script is the center of your project automation universe. The more you know about how to automate your builds, the easier your life will become and the faster you’ll be able to get work done.
We have a configuration management (CM) department that’s responsible for maintaining our builds. We aren’t allowed to modify the script ourselves. What do we do?
You need to be able to update your scripts continuously to meet your specific needs. It’s unlikely that anybody can be more responsive to your needs than you are. If the CM department is a bottleneck, ask your project manager for help. He may be able to give you control over the scripts.
Alternatively, you might be able to use a two-stage build in which you run your own scripts privately before handing over control to the CM department.
How do we find time to improve our build?
Improving your build directly improves your productivity and quality of life. It’s important enough to include in every iteration as part of your everyday work. The best way to do this is to include enough slack in your iteration for taking care of technical debt such as slow builds. If a particular story will require changes to the build script, include that time in your story estimate.
Should we really keep all our tools and libraries in version control?
Yes, as much as possible. See Version Control” earlier in this chapter for details.
Does the build have to be under 10 minutes? We’re at 11.
Ten minutes is a good rule of thumb. Your build is too long when pairs move on to other tasks before the integration cycle completes.
We use an IDE with an integrated build system. How can we automate our build process?
Many IDEs use an underlying build script that you can control. If not, you may be better off using a different IDE. Your other alternative is to have a separate command line–based build system, such as Ant, NAnt, or make. You risk duplicating information about dependencies, but sometimes that cost is worthwhile.
We have different target and development environments. How do we make this build work?
If possible, use a cross compiler. If that doesn’t work, consider using a cross-platform build tool. The benefits of testing the build on your development platform outweigh the initial work in creating a portable system.
How can we build our entire product when we rely on third-party software and hardware?
Even if your product relies on yet-to-be-built custom hardware or unavailable third-party systems, you still need to build and test your part of the product. If you don’t, you’ll discover a ton of integration and bug-fixing work when the system becomes available.
A common solution for this scenario is to build a simulator for the missing system, which allows you to build integration tests. When the missing system becomes available, the integration tests help you determine if the assumptions you built into the simulator were correct.
Missing components add risk to your project, so try to get your hands on a test system as soon as possible.
How often should we build from scratch?
At least once per iteration. Building from scratch is often much slower than an incremental build, so it depends on how fast the build is and how good your build system is. If you don’t trust your build system, build from scratch more often. You can set up a smoke-testing system that builds the project from scratch on every check-in.
My preference is to reduce build times so that incremental builds are unnecessary, or to fix the bugs in the build system so I trust the incremental builds. Even so, I prefer to build from scratch before delivering to customers.
With a good automated build, you can build a release any time you want. When somebody new joins the team, or when you need to wipe a workstation and start fresh, it’s a simple matter of downloading the latest code from the repository and running the build.
When your build is fast and well-automated, you build and test the whole system more frequently. You catch bugs earlier and, as a result, spend less time debugging. You integrate your software frequently without relying on complex background build systems, which reduces integration problems.
Every project should have a good automated build. Even if you have a system that’s difficult to build, you can start chipping away at the problem today.
Some projects are too large for the 10-minute rule to be effective. Before you assume this is true for your project, take a close look at your build procedures. You can often reduce the build time much more than you realize.
If the project truly is too large to build in 10 minutes, it’s probably under development by multiple teams or subteams. Consider splitting the project into independent pieces that you can build and test separately.
If you can’t build your system in less than 10 minutes (yet), establish a maximum acceptable threshhold and stick to it. Drawing this line helps identify a point beyond which you will not allow more technical debt to accumulate. Like a sink full of dishes two hours before a dinner party, the time limit is a good impetus to do some cleaning.