CHAPTER 12

• Monday, October 13

Over the past week, it’s clear to Maxine that Data Hub has figured out how to deliver better value, sooner, safer, and even happier. But it’s also clear that a new constraint has emerged. The constraint used to be getting environments—no one could ever get one, and when they did, it was never quite right. Then the constraint became testing, which started only when Development was finished with all their features; finding and fixing defects would take weeks instead of the hours or days it takes now.

Now it is obvious that the constraint is deployment—they are now able to quickly get features production-ready, but they still have to wait weeks for Ops to deploy their code into production.

Figuring out how to get Data Hub into production more quickly is no longer an academic concern. Tom is standing in the front of the conference room with the rest of the Data Hub team. He says, “Maxine, the suspicions that you had while you were out sick were right on target. According to Maggie and all her product owners, creating effective promotional bundles is one of the most critical and urgent Phoenix priorities.

“Kurt, the meeting we have with Maggie is scheduled for tomorrow, and you asked me to study up on this beforehand,” Tom continues. “Here’s what I’ve learned: Marketing is constantly experimenting with promotion campaigns to accelerate sales, and this is incredibly important as we approach the holidays, our peak sales season. For example, now that it is snowing in many areas, they want to create a winter promotion bundle: tire chains, ice-melting salts, and window scrapers. They also need to create a discounted price for that bundle, say twenty percent off. They also do promotions to customer segments—if you buy lots of windshield wipers, you may get offered a bundle of wiper fluid and glass defoggers, knowing that you may only need the smallest nudge to buy.

“Conceptually, it sounds easy. But here’s all the insane steps they need to go through to get this done: First, every new product bundle needs a new SKU, just like every other item we sell. These SKUs are used by almost every application in the business: inventory tracking, in our supply chain, our in-store registers, our e-commerce sites, even the mobile apps …

“We only create new SKUs in large batches every six weeks. After the SKUs are created, we also need to push all the application and business logic changes for these new SKUs. These are pushed to every application that needs to know about them. That’s often scores of back-end and front-end applications across the enterprise. You might have seen these go out at eight on Friday nights. And when that’s done, sometimes we even need to manually refresh certain production databases.

“Here’s problem number one: we only create new SKUs every six weeks, which is way too slow. Thanksgiving is a month away, and we’re already in danger of not getting those product bundle SKUs created in time.

“And the real truth is that it often takes us much longer than six weeks. We need to change so many applications during those pushes that if anything goes wrong during testing, the entire release is canceled … You can’t have new SKUs out there when some of the applications don’t know how to deal with them. There’s just not enough time to fix these things during the test period, so it’s all-or-nothing.

“And on top of that, Promotions also needs to rapidly experiment and iterate to discover which bundles customers are actually interested in and what specific factors lead to an actual purchase. Right now, iterating only once every six weeks is not fast enough to learn and adapt—our e-commerce competitors are doing multiple experiments per day,” Tom concludes.

“Wow, that’s really incredible,” Maxine says, looking at all the boxes he’s drawn on the whiteboard. “This is so much like the Phoenix architecture, which makes it so difficult for any team to independently develop, test, and deploy value to our customers. The architecture that supports the Promotions value stream that you’ve just drawn on the whiteboard shows how it’s almost impossible to move any work quickly to where it needs to go.”

She gestures at his diagram. “At every step, it’s entangled with so many other value streams. We have to synchronize with everyone else’s release schedules. If any of them can’t be released, then we can’t be released … It’s just crazy.”

“It really is. It’s frustrating that Data Hub is so tightly coupled to Phoenix and the BWOS,” Tom says.

“What’s a BWOS?” Maxine asks.

“Oh, that’s what we call the … you know, the big wad of … umm, crap. You know, the hundred-plus applications we connect to,” Tom says.

Maxine laughs. “I really think if we could deploy Data Hub changes into production on demand and fully decouple them from the Phoenix release schedule, we’d be so much better off … That way, if we have to cancel a release, we could at least try again the next day. With some practice, I bet we could get SKU creation down to one or two days.”

“I definitely agree,” Tom says. Maxine smiles, satisfied that they’re on the right track. And the value of doing this will be huge, she thinks.

“This may not be related, but I think it’s worth mentioning,” Tom says. “We have other huge problems being connected so closely to Phoenix. It sometimes sends us tons of messages that hammer the back-end systems that we connect to. We routinely see massive waves of transactions that cause huge reliability and throughput problems, and sometimes even data integrity problems. Sometimes it’s Data Hub crashing, but most of the time, it’s the systems that we’re calling that are the ones crashing.”

Cranky Dave piles on. “Dealing with those systems of record are a huge a pain in the ass. We don’t have any real API strategy around here. No one knows what APIs are available, and even if you do, no one knows how you get access to them or deal with their crazy authentication or pagination schemes. Everyone’s documentation is crap, and some of these teams don’t even care if their APIs don’t work as advertised.

“And once you do get someone’s API working, they’ll break it however and whenever they want, especially since they probably don’t version their APIs. So transactions start failing for our customers, and they blame us,” he continues. “They never give you all the data you need, so when you actually have an API change you want, you have to go through all these committees to get them approved!”

“It’s enough to drive anyone crazy,” Cranky Dave says, exhausted.

“We can definitely stop this madness,” Maxine says with certainty.

As promised, the next day, Kurt, Maxine, and the Data Hub team meet with Maggie. As usual, Kurt introduces all the Data Hub team members to Maggie and then asks Maggie to introduce herself.

“Many of you already know me,” she says with a smile. “My name is Maggie Lee. I’m senior director of retail program management. What that really means is that I have the P&L responsibilities of all the products and programs behind our stores, which includes physical stores, e-commerce, and mobile. My group of product managers own strategy, understanding the customer and market, customer segmentation, identifying which customer problems we want to solve, pricing and packaging, and managing the profitability of everything in our portfolio.”

She continues, “We bridge the business goals and everything that’s required to actually achieve them. That includes business operations, business analysts, and product managers, who work with Chris’ technical teams. I also have all the operational pieces required to deal with Sales, Finance, and Operations on my team.

“When Kurt said that you had some ideas on speeding up how we create promotion product bundles, you certainly got my attention,” she says. “Sorry I couldn’t meet even earlier, but as you can imagine, we’re all buried with a million things right now. It’s definitely a make-or-break quarter for us. For all of us.”

Maxine is already impressed. Maggie is in her mid-forties and has an unmistakable intensity about her. She is the same height as Maxine and obviously competent. She’s a no-nonsense type and always has a serious expression on her face. Maxine suspects that she’s Sarah’s forebrain, handling the million things required to keep a billion-dollar retail operation going.

Kurt explains what they’ve been working on.

Maggie looks at Kurt. “So you’re telling me that you could enable Marketing to create promotions entirely self-service, like our e-commerce competitors can? And that other changes could potentially be pushed into production on the same day?” Maggie says. “Holy shit, folks. If you can actually do what you say, this might be the miracle we’ve been hoping for. I’m not prone to overstatements, but I’m not kidding when I say that this could potentially save the quarter. And maybe even the company.”

Maxine smiles. “From everything we’ve studied, it’s clear that it’s way too difficult and takes too long to get those promotional bundles created. We’d love to fully empower your teams to do what it takes to create new promotions anytime you want and have them pushed out to all your sales channels within hours. There’s a lot we don’t understand, but conceptually, we should be able to do it. We just wanted to explore whether this would be valuable to you.”

Maggie nods. “Hugely valuable. Look, Steve has promised all the analysts that this holiday season we’re finally going to see an uptick in revenue. This is after years of over-promising and under-delivering. Everything hinges on Promotions being able to move the needle on sales. If you really think you can make this happen, we’ll do whatever it takes. What exactly is in the way?” she asks.

“Who isn’t in the way?” Kurt laughs. “We’re meeting with Information Security tomorrow, who could kill this effort on a whim. But the real threat is the TEP-LARB. We’ve put together a team to create our proposal, but people usually wait six to nine months to get in front of them,” Kurt says. “Unless, of course, there’s an urgent business need with a powerful sponsor.”

Maggie finally smiles, in not an entirely kind way. “For this, I think we need to bring in the big guns.”

“Who’s that?” asks Maxine, curious who could possibly be a more powerful sponsor than Maggie.

Maggie grins. “Sarah. Take it from me, there is no one more effective at busting down inconvenient barriers than she is. She’s like a chainsaw, great at cutting down trees.”

“… and sawing off hands,” Kurt mutters under his breath.

The next day, Kurt and Maxine meet with Ron, the security manager that Shannon introduced them to. They walk into the conference room and see that Shannon has arrived early.

“There’s no way I’d miss this,” she says, smiling. “I should have brought popcorn.”

Ron, who is in his mid-thirties, comes in and sits down. After introductions, Kurt walks him through their idea to decouple Data Hub from the rest of Phoenix.

Ron says, “Interesting idea. I remember when Data Hub was still called Octopus. Why the need for such a big change? It seems to be working well enough now.”

Kurt walks through all the reasons, and to Maxine’s surprise, Ron nods agreeably. This is going better than I thought it would, Maxine thinks.

“That’s exciting,” he says, agreeably. But then he takes off his glasses and puts them on the table. “Look, I really want to help, but I can’t. I’m responsible for making sure applications in my portfolio meet all applicable laws and regulations and that all those applications are secure. Given how radical of a change you’re making, I’m afraid we need to perform a complete due-diligence effort. And you simply can’t jump the entire queue. You have twenty people ahead of you who would scream bloody murder,” he says.

“But the Promotions capability is one of the most important features inside Phoenix, which is the most important initiative for the company,” responds Shannon. “Surely you see that ours should have higher priority, right?”

“Yes, but …” Ron says, shaking his head. “I don’t set the priority or order of the applications. That comes from the business. You know, our customer.”

“But we are ‘the business!’ And those ‘customers’ you’re talking about aren’t our customers—they’re our colleagues! Our customers are the ones who actually pay us money!” Shannon says, bright red with exasperation. “Everyone knows what the top goals are. The top priorities are what Steve always talks about in all the Town Halls. What else is more important than getting Data Hub successfully decoupled from Phoenix, so that the Promotions team can meet the holiday sales goals?”

Ron shrugs his shoulders. “If you want to change the order, you’ll have to talk to our boss, John.”

Kurt closes his laptop, clearly concluding that there’s nothing to be gained in this meeting.

“Fine, fine, fine,” Shannon says, resigned as well. Then she turns up the charm, saying, “Hey, could you at least give us all of the testing procedures that you’ll use to certify Data Hub, along with a list of tools you use to scan it? We’ll do our best to replicate it in our automated test suite. Maybe we can generate security audit reports for you on-demand.”

“That’s a great idea, Shannon,” he responds. “Come to my desk and I’ll show where all the documentation for the previous audits are.”

Maxine loves how Shannon takes every opportunity to get people on their side.

Watching them leave, Kurt looks at Maxine, shrugging his shoulders. “Could have gone worse, I suppose. Maybe we’ll fare better with the LARB.”

Maxine sighs. She wonders what is required to generate a true sense of urgency. When her dad had a stroke two years ago, she had remarked on all the bewildering processes in the hospital to one of her doctor friends. Her friend responded, “You were lucky. The processes in a stroke ward tend to be superb, because everyone knows that every minute counts and waiting could be the difference between life and death.

“The worst systems tend to be in mental health and elderly care, where there is less urgency and often no patient advocate,” she had said. “You can get lost in the system for years. Sometimes even decades.”

Maxine remembers what it felt like to be the patient advocate for her dad, doing whatever it took to get him through the healthcare system. Now, she recommits herself to doing whatever it takes to get her teams through the company bureaucracies—the Data Hub team’s sense of mission and urgency deserve no less.

Relentless optimism, she reminds herself.

As Maggie promised, they are on the LARB agenda on Thursday. Maxine is amazed and wonders what strings Sarah must have pulled to get them in so quickly. Then she wonders what Maggie had to do to convince Sarah.

Although Maxine recognizes the political necessity of pitching the LARB, she still resents all the time the team spent filling out the TEP—engineers should be writing code, not filling out forms.

It had many valid questions about architecture and security, but some questions seemed dated, reminding her of TOGAF architecture diagrams from decades past, clearly written for a different era: software development and testing phase gates, datacenter specifications, HVAC specifications, Check Point firewall rules (if applicable, of course) …

The Data Hub gang responsible for putting the proposal together is all here, sitting in the back of the room: Tom, Brent, Shannon, Dwayne, Adam, Purna, and Maxine. At one table sit all the senior Dev and Enterprise architects, and at the other table sit all the Ops and Security architects. They are all close to Maxine’s age, but mostly white males with a couple of Indian and Asian males in the mix. Maxine notices there’s not a single woman at either table.

Data Hub is second on the agenda. First up is a group pitching to re-platform all of their applications from a commercial product onto Apache Tomcat, a battle-tested and fully open-source Java application server. A younger woman confidently presents their case, which she delivers in a very thoughtful and competent manner. But when Maxine hears that all they’re looking for is permission to use Tomcat, she’s aghast.

Having to ask permission to use Tomcat in production is like asking permission to use electricity—maybe it was once considered dangerous, but now it’s commonplace. Worse, it’s apparently their second time pitching the LARB. Maxine’s heart sinks. If Tomcat is considered risky and controversial, their Data Hub proposal is going to get laughed out of the room.

After twenty minutes of skeptical questions from the LARB, the young engineer throws up her hands in exasperation. “Why are we so frightened of running software we wrote? We’re a manufacturing company. We wrote our own MRP and we run it ourselves. And for Tomcat, we don’t need to rely on a commercial vendor anymore. Some of the largest companies in the world use it. We’d not only save the company hundreds of thousands of dollars a year, we could finally do things that our current vendor won’t allow us to do. There’s so many capabilities we need to better serve our customers.”

Maxine gets goosebumps—not because the presenter mentioned her old MRP system, but because the presenter is clearly a brilliant engineer, fearlessly doing what she thinks needs to be done and not afraid to run things in production.

While the young engineer fields more questions from the LARB, Maxine texts the Rebellion in the chat channel:

Who is this engineer presenting? She’s awesome! She’s obviously Rebellion material. We should recruit her.

Adam texts back:

That’s Ellen. She’s one of the best Ops people around

Everyone nods at Maxine, agreeing with Adam’s assessment.

Brent adds in the chat channel:

Agreed. I had no idea she was working on this. This is great!

Maxine looks up when she hears Dwayne talking. “You have got to be kidding me. We created TEP-LARB to help evaluate new technologies. Apache Tomcat was created decades ago, and it’s either the second or third most widely used application server out there. If we aren’t brave enough to run Tomcat, we should get out of the technology game once and for all. I vote yes. And if you don’t, I think we all need to hear why.”

Someone from the Ops delegation says, “I don’t have anything against Tomcat. I’m just not comfortable with our ability to support this given our current staffing levels. We’re stretched thin as it is, and while I appreciate that this technology isn’t bleeding edge, we still need people to operate and maintain it …”

Dwayne interrupts, “But you just heard Ellen say that her team is willing to support it!”

Not even acknowledging Dwayne’s comment, the Security architect joins in, “And there’s the security risks. I’d like to get a historical report of Tomcat vulnerabilities, how quickly patches were made available, and any reported problems in patching. Maybe then we can come to a decision.”

Dwayne mutters, “For crying out loud. Ellen is the person who would write the security and the patching guidelines.”

“Thank you for your proposal. We look forward to this team presenting the requested information at our next meeting,” the Ops architect says, not looking up from the note he’s writing.

At the front of the room, Maxine sees Ellen and her teammates slump in exasperation. Ellen closes her laptop, nods respectfully to all the assembled architects, and takes a seat at the back of the room.

Maxine gives Ellen and her teammates the most enthusiastic thumbs up she can manage.

“Next up is Maxine and Adam on the proposal to move Data Hub into a new environment, running on containers, with automated code builds, tests, and deployments?” the Ops architect prompts.

Adam stands up, but after seeing the last presenter, Maxine already knows they’re sunk. No matter how well prepared they are, they’ll never be able to convince the LARB.

“… and to summarize, the urgent needs of the Promotions team requires us to get Data Hub functionality more quickly to our internal customers. We need a radically different way to store and retrieve data that allows us to be decoupled from the rest of the Phoenix teams,” Maxine concludes. “We’ve found a set of technologies that can help us achieve that, which have been battle-tested and used in production for over a decade at some of the largest internet properties on the planet: Google, Netflix, Spotify, Walmart, Target, Capital One, and many more. Based on our trials over the past several weeks, we are confident in our ability to support it, and we’re willing to support it ourselves if necessary.”

Brent, who joined them at the front of the room, adds, “The team supporting Data Hub production would be some of the most experienced people we have in Ops. Personally, I can’t even describe how excited I am by this effort. I think these technologies have applicability far beyond Data Hub and could really improve things for almost every application we support. We are willing to be available and responsible to resolve anything that goes wrong. Utilizing these techniques will help every Dev and Ops person at Parts Unlimited.”

Maxine sees Kurt smile at the team from the back of the room. Maxine is proud of everyone. It was a solid presentation. She sees Ellen grinning wildly, obviously impressed. But Maxine knows that it’s all for naught. The LARB was designed to be an organizational immune system to prevent dangerous changes—they are just too powerful and conservative.

Dwayne tries to rally support. “The LARB should foster innovative efforts like this, picking technologies that can help us win in the marketplace. We used to set the industry direction, making bold choices that left our competitors in the dust. People laughed when we created our own MRP system, saying we were idiots, but history has shown that that was the right thing to do. We were the first company in our industry to use thin clients in our factories, and because of that and hundreds of brilliant technology decisions, we became one of the most efficient and effective manufacturers in the country.”

Maxine looks around the room and sees some stirrings of excitement and renewed curiosity among the Dev architects. However, she sees all the Ops and Security architects shaking their heads. One of them says, “Dwayne, I appreciate what you’re saying, but we’ve never done anything even remotely similar to this. It’s embarrassing that we can’t even support Tomcat—but that shows you exactly why we can’t possibly support this. Unless there’s a group willing to volunteer to support this initiative as a side project, I think we need to table it.”

Dwayne speaks up, “Hell yes, I volunteer. And I’ll grab some people I know who would love to help the Data Hub team with the support responsibilities.”

“I’d love to help,” says Ellen from the back of the room. “I’ve been using Docker and the other tools you’ve mentioned for years. These are competencies we need at this company.”

“You’re in,” Maxine says to Ellen, smiling.

The Ops Chair looks surprised but says, “I appreciate your enthusiasm, but I’m afraid that we cannot support your initiative at this time. Let’s pick this up in six months and see if conditions have changed by then.”

Hearing enough, Kurt stands up and addresses the room. “Didn’t you hear the business context? Both Maggie Lee and Sarah Moulton have clearly stated that the company’s survival depends on this. This is so important that if you can’t support it we’re going to have to support it ourselves in Development.”

“We hear business people say things like that all the time,” the Ops Chair says. “We invite you back in six months to discuss it again. And now to other matters …”

Defeated, the team leaves the meeting, reassembling in a nearby conference room that Kurt booked in advance. Maxine invites Ellen and the three other engineers who presented the Tomcat proposal.

“Wow, that was so great. Are you really going to go rogue and run all this yourselves?” Ellen says, smiling ear to ear, not affected by the glum faces all around her. “If so, count me in. I’m Ellen, by the way,” she says, extending her hand to Maxine and then introducing her team.

“Good seeing you again, Ellen,” Adam says with a big smile. “Welcome to our merry band of rebels. If I’m reading the tea leaves right, I think we’re going to need your help soon.”

Ellen smiles. “The fact that you have Brent onboard is enough for me. What you presented was amazing. I had no idea anyone was working on these types of things here at Parts Unlimited.”

Brent smiles modestly, “But we still got our asses kicked, right?”

Kurt says, “We did indeed. But if all goes according to plan, by the end of the day there will be a memo going out from Chris and Sarah announcing a small re-org that will allow Data Hub to operate outside the conventional Ops and QA processes. That will be the official go-ahead to do whatever we need to do.”

Everyone on the Data Hub team cheers, surprised at the good news. Maxine hears Ellen mutter, “Wow. That’s some pretty powerful mojo you have on your side.”

Brent mutters back, “You have no idea. I’ll tell you later.” Adam laughs in agreement.

While almost everyone is celebrating, Dwayne is glum. When Maxine asks why, Dwayne says, “I just can’t believe the LARB didn’t support these efforts. We let you all down. What was supposed to happen was that they would see the grave danger on the horizon. They were supposed to support our cause. They were supposed to help … Like Gandalf getting the support of the White Council in Lord of the Rings …”

Maxine is surprised when Dwayne puts both of his hands on his temples, groaning. After a minute, he finally says, “But it didn’t work out that way at all.”

Brent laughs. “You’ve got it wrong, Dwayne. The Fellowship of the Ring wasn’t ever officially sanctioned by the White Council. Gandalf warned everyone that the One Ring was at large, but Saruman refused to help because he was already working for the evil Sauron. So, Gandalf went rogue. He went it alone. Just like we’re going to do.”

“Damn right,” says Kurt. Turning to Ellen and her team, he says, “You all doing anything after work? There’s a bar that we go to …”

“What the hell have you gotten me into?!” Chris says, fuming at Kurt. “Maggie and Sarah tell me that you’ve proposed to create your own Ops organization inside of Dev?! And that you’ve gotten some sort of exception waiver to start running some new Tier 2 services in the cloud?! I don’t suppose you ever thought to ask me first?”

Maxine is in Chris’ office with Kurt, Dwayne, and Maggie. Chris is clearly not happy, but Maggie goes to extraordinary lengths to describe the business outcomes that need to be achieved and the grave consequences of not doing so.

Chris stares out his window for several moments and then turns to Maxine. “Do you think we really have the chops to keep all this from blowing up in our faces?”

“Absolutely, with the help of Dwayne and Brent from Ops,” she says with certainty. “I’ll do everything in my power to make sure things go smoothly. I really think we’ve got this, Chris. And I promise to take the blame for anything that happens.”

At the mention of Dwayne and Brent, a pained expression appears on Chris’ face. He looks at her, obviously thinking, What about ‘don’t rock the boat’ and ‘stay in your lane’ do you not understand?

Maxine shrugs. She knows that Chris supported mission-critical services early in his engineering career, over twenty years ago. But ever since then, he’s only been responsible for the code, no longer running the actual services that it enables. Maxine could almost see him tabulating all the inconveniences this could create, all the things that could go wrong, balancing it against what could happen if he refuses.

“Fine, fine, fine. I’ll do it,” he says reluctantly. “You people are going to give me a heart attack,” and then shoos them out of his office.

As promised, Chris sends out a memo to everyone announcing a re-org—the Data Hub team is now reporting directly to him, and as an experiment, they’ll be exempt from the normal rules and regulations around changes, able to test their own code, deploy it, and operate it in production themselves.

“The email just went out,” Kurt says, grinning wildly. “We’re in the deployment and operations business!”

“Wow, that’s incredible,” Maxine says, still staring at the email on her phone. “You know, despite everything we did, I was pretty sure it wasn’t really going to happen.”

Kurt laughs. “I don’t think Chris had that much choice in the matter. Both Maggie and Sarah took this all the way up to Steve.”

With the Data Hub re-org, the team is now committed. They are working furiously to automate the production deployments and to figure out how to do production operations without centralized Ops. To what extent they needed to really divorce themselves from Ops for things like backup was still unclear and being negotiated.

The enormity of the challenge is exhilarating. The goal is clear: enable fast and safe deployments into production, and for the first time in years, do it using the same environments across Dev, Test, and Production. And everyone wants to prove that they can get everything up and running before the rest of the Phoenix Project even finishes their testing cycle.

Once again, they are in an imaginary race against the lumbering Phoenix Project.

Maxine is working with Dwayne, Adam, Shannon, and Brent, making slow but sure progress on getting the Data Hub production services to run on something besides the fastest bare-metal servers that money can buy … a decade ago. Many things in Data Hub blew up when installed on a current OS version … from this decade. They found several binary executables that no one could find the source code for. Data Hub had become this fragile and irreproducible artifact. That’s great if you’re an art collector, Maxine thinks, but utterly unacceptable when you’re running a mission-critical service.

They work methodically to create a Test and Production service that behaves like the old one, but can be spun up instantly in a container. For days, she’s mired again in the messy world of infrastructure, dealing with Makefiles, YAML, and XML configuration files; Dockerfiles; purging secrets from their source code repositories; and using all her experience to speed up build and test times. This, unfortunately, required lots of Bash scripts.

Maxine remembers a quote from Jeffrey Snover, the inventor of PowerShell. He once said, “Bash is the disease you die with, but don’t die of.” Maxine shares this sentiment. Infrastructure is messy work, almost the opposite of the pure functional programming she loves—in infrastructure, almost everything you do has a side-effect that mutates the state of something in the environment, making it difficult to isolate and test changes, as well as diagnose problems when things go wrong.

But she knows how important this work is, and every bit of knowledge and expertise that she can put into these environments and CI/CD platforms will elevate the productivity of every engineer at Parts Unlimited.

Looking around, she realizes that now some of the best engineers in the company are working on making everyone else more productive. That’s the way it should be, she thinks.

By the next Thursday, Maxine is thrilled at how much they’ve been able to accomplish with all the restrictions lifted. But something strikes Maxine as odd. She notices that all the Data Hub engineers are pitching in. She certainly appreciates their help and she knows that Project Inversion was supposed to disallow feature work, but still, there’s almost always some urgent feature that needs working on.

Suspicious, she asks Tom what’s going on. He says, “This sounds strange, but technically there isn’t any feature work even ready. Believe it or not, every feature is waiting on something from Product Management,” he says. “It’s everything from a customer requirement that needs clarification, a question about a wireframe, a choice that needs to be made between different options or priorities … Sometimes it’s something small, like where a button should go. And sometimes it’s something big, like them not showing up to the demo to validate what we’ve built.” Tom laughs. “They think we’re the bottleneck, but we’re always waiting for them.”

“Can you show me?” says Maxine. None of the things Tom described sounded good, but the part about the product manager not showing up for the demo pisses her off. What a disrespectful thing to do to engineers who built what you asked them to.

She watches as Tom pulls up a tool she hasn’t worked with before, this one used by the product managers to capture ideas from customers: the ideal customer journey, value hypotheses, manage experiments, and so forth.

“What are all those blue cards?” she asks.

“Good eye. That’s exactly the problem,” he says. “Those are all the features that we’re working on, but we’re blocked because of something we need from Product Management. Like all those reasons I mentioned before. Oh, and here’s some yellow cards which are the features we’ve completed but that haven’t been accepted by the business stakeholders yet. This one has been waiting for forty days.”

Maxine feels her face turn red, indignant that as much as Product Management complains about the need to get features to market quickly, all these blue and yellow cards represent where they are in the way, not Development. How can we keep Product Management accountable? Maxine thinks. Time to bring in Kurt.

Ten minutes later, Kurt is with them, staring at the sea of blue cards. “I get it. This is not good, but I have an idea,” he says. “By the way, did you know that Sarah put a huge design agency on retainer, and now they’re flooding some of the other teams with wireframe diagrams that will probably never get worked on? And no matter how much the Dev managers ask them to stop sending wireframes, they still keep coming.”

“Why?” Maxine asks.

“I think it’s because Sarah needs to show off the apps she wants to build,” he says. “But what’s funny is that when the designers came here, the last thing they wanted to do was wireframes. They wanted to learn about our customers, and they did a bunch of exercises to better clarify goals around the personas we used. There was even one session where we all drew wireframes,” Kurt says, laughing.

Working with designers fascinates Maxine. Early in her career, the ratio of UX and designers to developers was 1:70. These days, great teams doing consumer-oriented products have ratios of 1:6 because it’s that important to create products that people love. Every consumer these days knows what a professional app feels like. Apps that don’t have great designers are often ridiculed as “enterprisey.”

She’s seen teams still waiting to be assigned designers, eventually making their own wireframes, HTML and CSS styling, and icons just to keep feature flow moving. These are the projects that teams are actually embarrassed to show other people, she thinks.

The good news is that Sarah got a bunch of great designers. The bad news is that she put them all where they weren’t needed and were actually slowing important development work down by flooding their backlogs with things that didn’t matter.

That evening after dinner, while her family played with Waffles, she opened up her laptop. Something about the sea of blue cards that Tom showed her earlier had been bothering her, and she’s determined to get to the bottom of it.

That sea of blue cards is a part of the tool that the product managers use to manage the funnel of ideas to achieve business outcomes. This is a process that starts long before a feature is created in the Dev ticketing system. She logs into that tool using the credentials that Tom gave her. Browsing around, she can see when ideas were first conceived and brainstormed and all the various phases until it becomes an approved feature.

She searches for the first feature that she worked on with Tom about extended warrantee programs. When she finds it her jaw drops. That feature was first discussed almost two years ago. It started off as a small feature but was rolled up into a larger warrantee initiative, which then had to be pitched to a steering committee. When it was approved, they wrote up detailed specifications, which were pitched six months later. Only then were they approved (a second time) and finally funded.

This idea bounced around in the marketing and project management organization for almost two years, and then turned into a super-crash priority feature that had to ship by the end of the year.

For something this important, we wasted almost two years, she thinks. In the ideal, they should have just assigned a team that included developers to explore the idea and build a solution together. Instead of one product manager working on this the entire time, we could have had five people working on it. And we could have been learning the whole time, Maxine thinks.

She wonders how much of this specification document that was written two years ago is now out of date.

She pulls up the Dev and QA ticketing system and copies some dates into the spreadsheet. She spends nearly ten minutes Googling around, trying to remember how to do date conversions and date arithmetic correctly.

She stares at the screen, shocked. She does the formula a couple different ways, but she still gets the same number.

She texts Kurt:

We’ve got to meet tomorrow. I have something to show you.

Maxine is with Kurt, Tom, and Kirsten in a conference room projecting her laptop on the screen. Everyone is staring at it in disbelief, which she totally understands. She’s been thinking about this number all night. “Can that actually be right?” Kurt finally asks.

“I’m afraid so,” says Maxine. Kurt looks over at Kirsten, who is still staring at the numbers.

“Only 2.5 percent of the time required to go from concept to customers actually using the feature is spent in Development?” she finally asks, the disbelief evident in her voice. She stands up and walks to the large TV screen to look more closely at the spreadsheet. “Where is all the time going?”

Maxine says, “Long before the feature ever gets to Development, it goes through the funding approval process, which often takes over a year. And then once the feature is created, most of the time isn’t spent in-work, it’s waiting for a product manager to respond to a question. It’s the Square again. Teams are spending too much time waiting for product managers to get them what they need …

“And then once they’re done with the feature, they’re waiting for QA and deployment,” Maxine says. “This is terrible. We’ve spent all this time hiring more developers, but they often don’t have things ready to be worked on. And when they do finish a feature, it takes forever to actually get things into production so that our customers can use it. And often the only feedback we get are the annual focus groups.

“We don’t have a fast value stream,” Maxine says. “What we’ve got is more like a stagnant value pond, full of scum, breeding malaria.”

“Time to call Maggie,” Kurt says.

That afternoon, Maggie comes up with an elegant solution. She decides to move the Data Hub product manager from the Marketing building to a desk right by Maxine starting Monday.

In the conference room, Maggie tells him, “You’re the bottleneck. Your top priority now is to make sure any questions that the technology teams have are quickly answered. Nothing else takes priority over that.”

He balks and then proceeds to describe all the other demands on his time. Talking with customers, helping sales with negotiations and trying to break them of bad habits, briefing internal executives, working with business operations, arguing with business stakeholders to agree on a product roadmap, escalating things up the chain to get approvals for urgent issues … And way down the list was answering questions from developers.

Maxine listens with interest, realizing that no one can get anything done when you’re pulled in that many directions. Maggie also listens patiently, nodding and occasionally asking questions.

When he’s done, she says, “If you’re too busy to work with the technology teams, I’ll move you into a pure product marketing role, and you don’t have to move your desk. Right now, I need product managers who are working side by side with the teams who are building what will achieve our most important business objectives. If you still want to be a product manager, I’ll figure out how to clear your plate and get those other responsibilities assigned to someone else.

“Don’t give me an answer right now,” Maggie says. “Think about it and let me know first thing Monday morning.”

Maxine is impressed. Maggie does not mess around, she thinks.

By mid-day Monday, that product owner moved his desk right next to Maxine. The dynamic immediately shifts. To get answers, things no longer wait on tickets. Engineers are able to just swivel their chairs around and ask him. Things that normally took days are being resolved in minutes. And better yet, engineers start gaining a much better understanding of the business domain.

Maxine smiles. The team of teams keeps growing, and it feels good.

From:			Alan Perez (Operating Partner, Wayne-Yokohama Equity Partners)
To:			Dick Landry (CFO), Sarah Moulton (SVP of Retail Operations),
Cc:			Steve Masters (CEO), Bob Strauss (Board Chair)
Date:			7:45 p.m., November 5
Subject:			Strategic Options **CONFIDENTIAL**

Dick and Sarah,

For our next meeting, I’ve asked an investment banker we’ve used in the past to brief us on the market outlook for the retail and manufacturing sides of the Parts Unlimited business. Could you present a high-level briefing on the Phoenix initiative so we can get their thoughts?

Given the criticality of the upcoming holiday sales performance, I thought it might be useful to introduce ourselves to them sooner rather than later. Hopefully any valuation estimates will be anchored before any disasters. (You never want to talk to bankers when you really need them. They can always smell fear.)

Sincerely, Alan