Running an enterprise-grade data program is no easy feat.
BigQuery lowers the barrier to entry. For free, you can create a data warehouse that runs 24/7 without maintenance: no operating system, no patching, no running out of hard drive space. That was the promise we explored in this book. However, this warehouse also starts out completely empty and valueless. Your empty warehouse is the first blank page of a book called Data Warehousing at Your Company with BigQuery, and you must write it yourself. Much of it will be literal documentation, code, and configuration. An equal part will be abstract. By forging the right relationships with your business stakeholders and constructing the right process for your organization, you will succeed at establishing your data program. You can then build on that success for the medium- and long-term vision.
There’s a relevant analogy in the comparison between software architecture and building architecture. When a building is finished, it’s there. When you go to the coordinates, you will be at that building. It may have structural flaws, like the Leaning Tower of Pisa, but it’s still the only building at coordinates (43.7229559, 10.394403).
No such guarantee for software! You can construct the most perfect algorithm or write the most brilliant code of all time, and it’s possible that no one will ever know. It would be as if you could build the Leaning Tower of Pisa in an extra-dimensional pocket and then “deploy” it to those coordinates. Of course, then you could also dismiss the engineers, patch the building, and redeploy it.1
This property of software creates an unresolvable tension between “getting it right” and “getting it done.” No value is added by constructing a masterpiece in an extra-dimensional pocket and leaving it there. The first challenge of your data program is to ensure its existence. Success rests at least as much on the people as it does on the technology. Thus, BigQuery projects can still fail. Maybe the complacency that comes with a painless setup makes them even more prone to do so—that’s definitely not a technology problem. The good news is that you have the necessary tools to surmount this problem for all three edges of the golden triangle.
You can and should read books about databases, books about organizational theory, and books about application management lifecycles. Through the individual examination of each facet, it becomes clear that the triangle’s central force is data.
Think about it: a process needs real-world data to understand and improve it. Personal conflicts often result from poor communication and conversely could be prevented with accurate, accessible data. (How many bar fights have been prevented by the Internet?) Then, technology makes it possible to grapple with these giant, global-size problems. You can’t sustain a modern data program with thousands of cuneiform tablets eroding in a cave.
In the end, the biggest catalyst is the interaction between humans and technology. As the data program takes off, it will begin to produce insights. Those insights will inspire people to do a thing a little better. That increased motivation and ability will manifest in data streaming into the warehouse. That improved data will produce even greater insight. And so the cycle continues.
With the addition of real-time data analysis, those insights will arrive at a faster and faster rate. Returns multiply. Sifting through all the data in your organization and beyond will spark new connections that no one dreamed of alone. The impossible becomes obvious. The obvious becomes ambient. The process of discovery produces unexpected answers and even more unexpected questions. And you sit at the center of it all.