Give me a condor’s quill! Give me Vesuvius’ crater for an inkstand! Friends, hold my arms! For in the mere act of penning my thoughts of this Leviathan, they weary me, and make me faint with their outreaching comprehensiveness of sweep.
—Herman Melville, Moby Dick
Drawing on my fine command of the English language, I said nothing.
—Robert Benchley
Meet the Razor
Whales, my friends, have a lot of blubber. If you’re going to do battle with a leviathan, you’ll have to trim some whale fat. Law of nature, sorry.
The Razor is the enforcer of bureaucratic parsimony. It’s wielded by us, Knights of Occam, the bureaucratic equivalent to Six Sigma Black Belts. Bureaucracy is a factory whose output is compliance. A good bureaucracy—like a good factory—is one that uses the least effort and lowest cost to produce its product and can produce it fastest. The Razor strives for minimal viable compliance—meaning the fewest number of constraints consistent with the company’s situation and risk posture—and strives to achieve its desired controls with as little whale bulk as possible. The Razor trims bloated bureaucracy into lean bureaucracy; ponderous whales into sleek and speedy marlin; trolls into trainers. A visit to the Razor provides a shave and a haircut that costs less than two bits and makes a whale spiffy and fun to swim with.
This next set of tactics is from the playbook of the Razor.
R1: Step Out of the Circle
The less risk, the less oversight that’s needed. We faced a vicious circle in the government: our programs were large and therefore risky; because they were risky, they needed more oversight; because they needed so much oversight, the oversight process was burdensome; because it was so burdensome, no one wanted to do it too often; because no one wanted to do it often, they loaded all their work into a single, monolith of a project, which made it riskier; because the project was risky …
Here’s another vicious circle: a project that seems to be running behind schedule attracts more “help” from overseers; the help requires burdensome meetings and additional work to report on the project’s status and the planned remediation; that additional work takes time away from project activities; the project goes further behind schedule …
One more: projects sometimes fail spectacularly; when they do there’s a call for additional oversight; the additional oversight mechanisms constrain future projects in ways that make them likelier to fail—or, if nothing else, add lots of costly overhead that reduces the return on each dollar invested; the poor return causes calls for additional oversight …
Perhaps you have a friend who has experienced one or more of these scenarios. There’s a way to break out of these vicious circles, and it’s simple: Only do small projects.
R2: Shrink Everything to Skunk Size
A smaller project is a less risky project, and therefore requires less governance ceremony. We surely understand that a multibillion-dollar, five-year government IT project will require very formal and extensive oversight. But a single DevOps team working for a month?
What could a single DevOps team accomplish in a month? A lot, if they can focus on producing results rather than writing eighty-seven documents and enduring eleven gate reviews. DevOps and the cloud provide leverage for a small team: they amplify its efforts and magnify its returns. Small government IT investments sounded crazy when we started down the path at USCIS. Of course, the point was rarely to end the work after a month but to reevaluate the investment and make incremental decisions to continue funding it. So, while the risk was only for one month, the vision was much larger.
In the government at least, differentiated controls based on spending levels are common. If we could hold our spending to less than $10 million and use our standard DHS contract vehicles, for example, then we wouldn’t have to do extra work to protect ourselves from vendor protests and would face a less burdensome set of procurement controls. If we kept our contracts to less than about $4 million, we could award them directly to disadvantaged small businesses without the overhead of the usual competitive process. At under $150,000 there was a simplified acquisition process available under the prodigious six-thousand-page FAR.
Businesses too are used to applying simpler controls to less risky spending. For example, for expenses less than a certain amount employees might be able to use a purchasing card rather than a formal purchase order. Even if the bureaucracy is not yet set up that way, tailoring administrative burden to the level of risk might be palatable to the enforcers of the spending controls.
To be clear: I’m not suggesting that you get around regulations by chopping a large project into projects that are just barely small enough to avoid oversight. I am suggesting that you minimize the scope of an investment, following the Agile law “Maximize the Amount of Work Not Done,” and that you break large projects into staged incremental deliveries, each of which stands alone as a valuable product. In other words, don’t just make projects smaller—make them smaller in a way that truly reduces risk, because that’s what justifies a simpler oversight process.
The cloud, DevOps, and other contemporary IT delivery approaches make it possible to work iteratively and incrementally. You can produce valuable business results in just days, and then build on them in small increments. We even broke down our largest initiatives—let’s say, modernizing the E-Verify system*—into small, specific, and concrete business goals (five in all), which we then assigned to small teams and invested in individually.
The ultimate of smallness is the skunkworks. A skunkworks is a below-the-radar team that gets stuff done fast with minimal administrative ceremony. A skunkworks should not be sneaky, though sometimes it works stealthily. Rather, it should be small enough—and therefore low-risk enough—that it appears so much like a rounding error that it escapes attention. It should require a smaller commitment of resources than the finest-grained decision-making threshold. You can “just do it” because it uses odds and ends of resourcing.
Ideally, the result of a skunkworks project is that you can now ask permission to conduct a project, using the argument that its risk is low because it’s already finished. Or at least you can show a prototype or proof of concept that is vastly risk lowering for a decision-maker.
For example, one of my employees at USCIS told me of an idea he called Network in a Backpack. He stopped by my office one day with a small daypack and dumped out its contents on a table. His team had stuffed this back-pack full of hardware to solve a business challenge that I was already familiar with: our refugee officers travelled to refugee camps around the world to interview refugees for admission to the US. These refugee camps were often in difficult locations—sometimes desert shantytowns where power and internet connectivity were inconsistent and where dryness and dust, bouncing on desert roads, and other physical hazards led to damaged equipment that we couldn’t replace quickly in time.
Network in a Backpack contained ruggedized equipment that would allow our officers, with no technical skills, to quickly build a network in a refugee camp to connect their laptops and printers together. They could then connect it to our fingerprint databases and other information back in the States. We’d be able to control and monitor it from Washington, DC. And it would meet our security needs, even assuming a hostile local government and terrorist recruiters in the refugee camp.
There was very little risk for me as a manager in supporting Network in a Backpack—because, well, it was already done. It solved a problem that I wanted solved but hadn’t yet dedicated any effort to. I’d had no idea that his team was working on it. They’d done it on their own initiative, knowing the business challenge and hoping to solve it creatively. It cost so little that I hadn’t noticed. It was a perfect skunkworks project.
R3: Shift Left
Bureaucratic workflows often require employees to stop and seek approval to continue to the next phases of a project, and very often require approval of the finished work product before it is used. In old-school IT, a formal SDLC (software development lifecycle) governs the stages and reviews of an IT delivery project. MD-102’s eleven mandated reviews were used to verify, among other things, that the project’s requirements had been fully documented before design work could begin, that design had been fully completed before building could begin, and, at the end, the big one—that the system had been fully tested and secured and that users were ready for it before it could be deployed.
DevOps brought us the concept of “shifting left”—moving these validation activities earlier in the delivery process (to the left on a Gantt chart) or, ideally, just absorbing the validation activities into the everyday flow of work. For example, instead of checking a system’s design in a design review to make sure it follows enterprise architecture standards, the architecture team now gets involved before and during software development. They might write their own reusable code that implements their preferred architectural patterns for the development teams to use. They might make their preferred open-source products easily available and coordinate different teams to make sure their code interoperates smoothly. The overall impact is that the system doesn’t need to be reviewed later for compliance, because it’s been observed to be compliant all along.
Shifting left doesn’t remove bureaucratic controls—on the contrary, it increases their effectiveness. It makes the bureaucracy lean and enabling. It’s leaner because shifting left removes sources of waste like rework and waiting time (waiting for the gate review). It’s enabling because the approved architectural building blocks become tools in the developers’ hands: they help them do their jobs better and help the enterprise architects stop being no-saying trolls. And those subject to the standards can question and participate in formulating them.
At USCIS, we also shifted left through training. In the gate-rich process we’d started with, QA testers reviewed each system before release to make sure it was fully accessible to people with disabilities, and rejected the code when they invariably found that it wasn’t. Instead, we trained the development teams on accessibility and how to test for it themselves. We certified “Trusted Testers” who could demonstrate their skills in accessibility testing, then required each development team to have at least one Trusted Tester. The effect was to make each team responsible for its code’s accessibility.
We discovered that the majority of accessibility flaws came from one or two programming mistakes (forgetting to include alternate text for images, for example). We gave the programmers automated test tools that would help them find those sorts of mistakes and quickly eliminate them.
The “shift left” strategy is broadly applicable. For example, it can be used to encourage the organization to agree in advance what conditions need to be satisfied for compliance (thereby increasing the rigor of the bureaucracy!). In the security example I gave earlier, by definition, if the code passes the tests, it is compliant. The tests could be updated any time security learned something new. They served, therefore, to memorialize best practices and all the organizational learnings around security.
We used a similar tactic to implement financial controls. Under MD-102, the overseers became involved only after a project “failed” by “breaching” its agreement to reach certain cost and schedule milestones. But in our revised oversight process, we gave the overseers a monthly analysis based on hard data and invited them to interfere. We shifted their oversight to the left and made it participative instead of punitive.
Shift left uses guardrails to create a framework within which delivery teams can be creative. As long as their code passed its security tests, software developers could move as quickly as they liked and try out any new ideas they came up with. And the automated policy enforcement tools in production provided an extra safety net to encourage developers to make changes, knowing that their risks were mitigated.
R4: Respect Diminishing Returns
IT is filled with situations where there are diminishing returns to incremental effort. The bureaucratic overhead of IT projects is also often subject to diminishing returns.
For example, one of our MD-102 documents, the System Design Document, was considered important because it helped when a contract ended and one contractor was replaced by another. In the interests of fairness and agility, we wanted to make sure a new contractor could pick up the work and quickly continue with it. So, the design document’s purpose was to teach the new contractor everything they would need to know to do so.
The problem is that there is more or less an infinite amount you can write about the workings of a system, and no matter how much you write, the new contractor will still not be fully up to speed. And they probably won’t bother reading all of it anyway, especially if it’s too long. Ultimately, they’ll need to look at the code itself to fully understand it. There are diminishing returns to documentation, in other words.
So, the standard of trying to bring the new contractor 100% up to speed through a design document didn’t seem right. We realized that the automated test scripts we were now writing would do some of the work, since the new contractor could learn what a given piece of code was supposed to do by looking at its tests. As for the design document, we decided that its goal would be to get the new contractor something like 60% of the way toward understanding the system. Because of diminishing returns, that decision vastly reduced the amount of documentation teams created. We also asked the software developers themselves to write the documentation, since they’d know best what information would be most useful to incoming developers. And to make it easier and faster for them, we let them write the documentation on a wiki that they could easily change.
The goal of getting a new contractor 100% oriented through a design document was causing waste and bloat in our practices; a simple acknowledgement of diminishing returns was the key to trimming it.
R5: Address Costs of Compliance
Bureaucracy, as we know, can be wasteful—it leads to delays and hidden costs. Those costs fall through cracks unobserved or unacknowledged because (1) people assume that no cost is too high when trying to avoid “failure,” (2) they are hidden in different budgets, and (3) they are largely personnel costs, and no one tracks exactly how knowledge workers spend their time. Because bureaucratic costs are hidden, they’re not well controlled; often they’re thought of as nondiscretionary spending. But that’s wrong—except when they’re absolutely required by compliance frameworks like SOX and HIPAA, they are, actually, discretionary.
Add up the costs of red tape. First look at lead times, then actual costs. The cost of writing all the documents for MD-102 was tremendous. Add the cost to get them all signed—feedback, rewrites, and approvals. The meetings required for assembling the documents, reviewing their status, and discussing them. The document reviews, the people attending, the time it took to get them all in the same room. The status meetings to check on the progress of each large project in progress. (As opposed to incremental delivery, where status is conveyed simply by what has been delivered.) The Gantt chart for one of our projects became so large and cumbersome that it required twenty full-time people just to update it.
There is some cost at which incremental controls become value destroying. What’s the right level of spending?
Advanced razor tip: Add up the time spent doing versus time spent watching as I discussed in Chapter 9. This is an advanced tip because it can make people feel threatened. Don’t cheat by only including the IT department. Doing includes things like writing code, testing, deploying, and engineering infrastructure. Watching includes things like project management, status meetings, management, writing (certain) documents, getting sign-offs, governance processes, compliance specialists, CMMI experts … probably most of your costs.
Once you know your cost of watching, you can make decisions about whether you’re doing the right amount of it. It will probably cause a monkey-shock and generate urgency. What’s the right ratio of compliance to productive activity? What’s the company’s actual ratio? Is it possible to reduce that cost of compliance without changing the amount of risk mitigation?
This ratio should matter a lot to us if we’re trying to cut costs. Sometimes, the cost of controls is the best place to begin trimming.
R6: Promote Transparency
Disasters are often public; the costs of guarding against them are not. This difference allows blubbery bureaucracy to flourish, perhaps even incentivizes it. But what if those risk-mitigation costs were public, or at least publicized within the organization, even if just at senior leadership levels? The fact that it took seven weeks to complete a balanced workforce assessment was a dramatic statistic that stimulated action.
What if everyone knew that it can take two years to complete the PRA process? Now that I’ve told you about it, aren’t you outraged? The ratio of watching to doing is another area for generating shock and awe. Do you feel good that your tax dollars are paying for watchers, not doers? That they’re writing about their “full doctrines” for facilities and materiel when they should be building software? Or that seven people must sign off on a document that says the workforce is balanced? Boards of directors and company shareholders might feel the same way.
These details lie buried in our processes. But between the Monkey and the Razor there’s plenty to surface that would outrage shareholders. Would they really be happy with the time it takes to make decisions? The amount of time spent in meetings?
R7: Apply Lean Techniques
We don’t often look at bureaucracy as a problem of leanness. But that’s precisely what it is, once you’ve divorced yourself from metaphysical pathos. Fortunately, the entire toolkit of Lean techniques is available to those who wield the razor. Here are a few types of waste that tend to accompany bureaucracy.
1. Waiting time: Some causes: Waiting for people who have to sign off on documents. SLAs within parts of the value stream, which cap the maximum amount of time the process can take, but also tend to determine the minimum time, as people work backward from their deadlines. Waiting until meetings take place (coordinating everyone’s schedules!). Waiting for security reviews.
2. Handoffs: Because bureaucracies tend to have a functional division of labor, work must pass between different teams, with each handoff adding overhead. For example, the handoff of requirements from the non-IT part of the business to IT adds tons of overhead—the requirements must be fully documented and signed off on, and additional requirements tend to be added because later scope creep will not be allowed.† The handoff to oversight bodies and steering committees for review tends to add plenty of waste, because those bodies are less familiar with details and day-to-day activities and have to be educated.
3. Rework: Because no-sayers and reviewers arrive late in a process and their job is not to improve the work but to return it for changes, rework tends to be maximized. If the pharaoh reviews the pyramid after it’s built and decides that he doesn’t like the view from his tomb—or wants more room for his harem in the next life—then the slaves will have a lot of rework ahead of them.
4. Overspecification: You know the story of the $3,000 military coffee pot from Chapter 1. If you write specifications for a product before you know how much each requirement will add to the cost, then you’re making decisions with incomplete information, and it will cost you. This often happens with requirements given to an IT organization—somewhere embedded in those requirements is a feature that is enormously but not obviously expensive—only IT will know it, upon deeper analysis, but not the requestor. In a one-way, bureaucratic process, where someone issues orders and someone else follows them, there are bound to be requirements that are costly but don’t really mean that much to the requestor.
5. Learning the rules: Ever add up the cost of training people on the rules, correcting them when they get them wrong, or interpreting the rules when their application isn’t clear? If the policy is that employees should only fly economy class, what happens when business class turns out to be cheaper? One employee after another will ask questions, and the guardian of the rules will spend time interpreting the rule. The War-and-Peace-times-five-length FAR takes time to read and understand, and the chance of making a mistake and needing correction is high. There are many training classes offered inside and outside the government on how to understand the FAR. The more bureaucracy, the more waste there is in learning the bureaucracy.
Those are just some examples—a bureaucracy that has not previously been subject to Lean treatment will undoubtedly be holding in plenty more waste. Although bureaucracy tends toward bloating (a law of bureaucratic entropy?), it doesn’t need to. Even within a bureaucratic structure, we can enforce lean practices.
User’s Guide to the Razor
By all means, look at your value streams and see where bureaucracy is adding to your lead times! Calculate its cost, in lead time and dollars, then find ways to reduce it while still mitigating the same risks and achieving the same results. Keep in mind:
E-Verify is a USCIS system that companies use to verify that their employees are eligible to work in the United States. -au. |
|
I’ve discussed these topics in detail in my previous books War and Peace and IT and A Seat at the Table. -au. |
|