The future of ecommerce analytics is challenging to predict, but it is bright and ever-evolving. As you’ve read in this book, there are many ecommerce subject areas and approaches to analysis, which future technical, academic, and scientific progress will continue to advance and expand. Although the future will inevitably lead to the creation of new technologies, architectures, tools, models, algorithms, and methodologies for advanced analysis, this chapter lists what could be on future road maps for ecommerce analytics. The chapter is framed less on how to analyze data. Instead, I present new themes, leitmotifs, concepts, and capabilities that will assist in infusing analytics across an ecommerce business.
By talking to practitioners and business leaders and from professional work, one can begin to see the signals of what the future may hold. We live in a cluetrain world where analysts are increasingly required to seek out and use data that exists in many different sources both inside and outside the company. The data isn’t only quantitative; it’s qualitative. Data volumes can be enormous, and many ecommerce companies have “big data.” This huge amount of data collected is not usually ready for immediate use; it needs to be cleaned and prepared, which is why data scientists spend inordinate amounts of time getting data ready to analyze. Meanwhile, the demand for analysis in ecommerce business continues to grow, and people have higher expectations than ever that data can help them. As a result, analytical projects are numerous and increasingly complex. Today’s ecommerce analytics projects require cross-functional team collaboration and the reuse of existing data and analysis. Increasingly analysts are, quite correctly, tasked with identifying financial impact. I hope you’ve gathered by reading this book that I strongly believe analysis done for businesspeople must be tied back to improving the financial performance and the health of the business. Meanwhile, there is an expectation not only that analysis will be descriptive, exploratory, explanatory, and inferential but that it is possible to predict outcomes and use algorithms to prescribe the best course of action to take. Although theoretically true, in order to do advanced analysis, you need to have the right data to do it. And companies sometimes don’t have the data or the data they think is applicable isn’t. To course-correct, analytics projects can be handled like IT projects, but that is not optimal. Analysis projects, unlike IT projects, may not produce an outcome that is expected; sometimes you have to experiment and fail before you find the right data to analyze or the right model and algorithm to use. Fortunately in analytics, like in life, failure can be the mother of success. Thus, collaboration and socialization of analysis is important to ensure the business, technology, and the analytics team are aligned on work, expectations for output, and what the desired outcomes for analytics could be, including the possibility to learn from failure. Meanwhile, analysts are required to use multiple technologies to do their work. No one tool does it all. In fact, there are hundreds of software tools and technologies for doing analytical work—from collection, to metadata, to processing, to data modeling, to data preparation and cleaning, to reporting, visualization, analysis, and data science. Future software capabilities must accommodate enabling analytical work in these disparate, global heterogeneous environments that contain assorted data types.
But we need to keep in mind that technology is not the panacea for analytics. People are the solution for advancing ecommerce analytics in such a fast-growing, rapidly evolving ecommerce ecosystem. In this ecosystem, there are companies of all sizes—from major brands, smaller companies, start-ups, and even individuals selling products online in most industries. Competition makes ecommerce a challenging industry in which to win customers for the long term. All but the largest brands are fierce competitors to each other using discounts and razor-thin margins to grow market share and a share of wallet. Ecommerce companies start up, try to build excellent and creative experiences across different devices, and either succeed or wither and then fail. The global ecommerce market is international, with ecommerce brands popular in some countries and unheard of in other countries. For example, Taobao in China lists 800 million products and has 500 million customers; Tmall in China has 181 million customers; Flipkart, Jabong, and Snapdeal in India are massive ecommerce companies (Joson 2015). There are small players selling specialized goods on sites they own, and marketplaces where multiple vendors offer their products and services, and then, of course, major players in ecommerce most people know as household brand names, like Amazon, eBay, Walmart, BestBuy, Alibaba. The future of ecommerce analytics must address global, large-company concerns and also the needs of medium, small, and individually owned ecommerce businesses.
As the future progresses, the ecommerce companies, user experiences, and even device form factors will change in new ways that need to be measured. For established companies that have physical stores, omnichannel analytics requires understanding customers wherever they shop both online and also back at the store. For example, there is a trend in being able to pick up online orders at nearby physical store locations. This merging of the offline and online customer experiences requires the analyst to consider new methods of fulfillment and how they impact path to purchase and the consumer mind-set. To compete with in-store pickup, online-only ecommerce providers are speeding ordered products into the home with extremely quick delivery—one hour or less shipping from Amazon is being tested right now. Traditional shipping processes in which delivery is made by vehicles owned by a shipping company could shift toward drones or even self-driving cars ready to deliver core products. All these new methods for fulfillment will need to be tracked and analyzed. Predictive delivery based on customers’ purchasing patterns for core goods may be offered as part of new subscription models. When ecommerce features are integrated into products that are connected to the Internet, new channels for ecommerce become a reality. The Internet of Things and the even more ambitious concept, the Internet of Everything, will create new behaviors related to products and the ability for customers to interact with the products, even buying replacement parts or related products directly from within the product. The ecommerce experience itself will become virtual when virtual reality begins to be mainstream; trying on and buying a shirt will never be the same with Facebook’s Oculus. The dynamic personalization of most aspects of ecommerce based on knowledge derived from behavioral, transactional, and other customer data will create new experiences that are truly individualized. These experiences will be augmented with virtual assistants that use conversation to drive commerce. You will be able to tell robots what to buy for you in the brave new world. Intent and context will be in deep learning and used by artificial intelligence to create new commerce opportunities both in the physical and virtual worlds. The future of commerce isn’t in Minority Report; it’s right now in the apps on your smartphone and in the pervasive Internet and home networking, and it’s being thought up as you read by innovative companies, including but not limited to Facebook, Google, and Apple.
To understand and make sense of all these potential new changes and advances in ecommerce experiences and functionality, a solid data and analytics foundation is of course required. This foundation must take into account privacy, security, governance, and management of data, while enabling analysis of behavior, marketing, products, orders, and customers within an analytical-driven and informed organization. The entirety of this book discusses and describes the what, why, and how of starting, building, and/or evolving your approach to ecommerce analytics. Following are ways to think about how to extend or build analytical capabilities for the future, categorized as follows:
• Data collection and preparation. New technologies that increase the efficiency and effectiveness of collecting, ingesting, cleaning, and unifying data will increase in importance and change the work done by analytics teams.
• Data experiences. Ecommerce experiences will be driven from data in ways beyond current personalization and collaborative filtering. These include guided buying experiences driven by Chatbots and artificial intelligence, augmented reality, virtual reality, wearables, the IoT, and the IoE.
• Future analytics and technology capabilities. These capabilities will change the role of the analyst, including the type of work analysts will do in the future, the emphasis of how tools enable people to do analysis, and the role of data mining, machine learning, and artificial intelligence on replacing and augmenting human work.
Analysts in 2016 are commonly asked to analyze different sources of data in order to answer a single business question. As I’ve described in this book, systems that generate and store data can exist inside and outside the ecommerce company. Traditional ways of dealing with all this data were to replicate some or all of it using operation data stores, ETL software, and data staging environments to create a data warehouse from which governed data could be extracted and used. As described in Chapter 13, “Integrating Data and Analysis to Drive Your Ecommerce Strategy,” the long durations it can take to load and extract data from traditional data warehousing technologies are causing companies to rethink their approach. The future could hold these possible realities:
• Ecommerce data lakes will rise in importance. Instead of storing data in raw formats that need to be extracted, transformed, and loaded in centralized databases, ecommerce companies will manage data by keeping it in raw formats within a data lake. Data will then be accessed from the data lake and enabled for business use by data storage and processing technologies such as Hadoop and Spark. Ecommerce platforms, of course, will still underpin and provide the operational foundation for ecommerce sites to work. But the volumes of data generated by the various components in ecommerce architectures beyond the platform, such as digital analytics, product databases, social media, audience data, and market research, will be pooled into data lakes, so the business is aware of the existence of the data and is ready to begin the process of preparing it for use. This could lead to ecommerce DMP (data management platforms) as a layer on top of the data lake.
• Data cataloging and data profiling will become a concept to consider for ecommerce. If all the data is stored in a data lake and contemporary big data technologies allow the analysts to work with it, then how does the analyst know what data is available? The answer exists in building capabilities for ecommerce analysts to create data catalogs of the data, which include data definitions, canonical queries, and data lineage back to the original source before the data laking. In addition, data that is available must be profiled in raw and curated states to make its content and other statistical measures understandable. In this way, data profiling helps an analyst understand whether the data in the data catalog is relevant and applicable to the analytical effort.
• Data preparation will be done by the analytics team. Currently it has been anecdotally claimed that 80% of a data scientist’s time is spent not on modeling, testing, refining, and operationalizing advanced analytics, but rather on cleaning data so that the data can be used. A related challenge upstream is that technology teams are often faulted for not having accurate data for business purposes, or data availability is latent and comes too late for business usage, or the data can’t be joined or unified in the desired way. In the future, ecommerce analytics teams will take, in the worst case, raw data, and in the best case, curated data, from technical systems that are governed. They will use data preparation tools to clean data, join data, and script the data cleaning and joining processes so that the prep work can be reused again and again on the same data set. Machine intelligence, data visualization, and learning algorithms will assist the analyst in finding missing values, integrity issues, and other data problems in huge data sets, in ways that just can’t be done in Excel or using SQL. No longer will analytics teams have to work with data of questionable quality or be reliant on IT or insufficient software that can work only in aggregate. The analytics team, instead, will have data preparation tools that enable them to access data, create a copy, and clean and prepare data in a systematic and reusable way suited to analysis and data science.
• New data sources will arise from which data will be generated and collected. The rise of the Internet of Things and the Internet of Everything will be embedded in consumer goods, from appliances to wearables, to augmented reality and virtual reality. This will of course change how customers interact with products. These interactions are behavioral data to be captured and shared, when chosen by the user, with ecommerce companies that can do something to use and act on the data. Waterproof shoes get wet? The sensor starts the warranty process. Temperature changing as detected by your wearable shirt? Render an experience in which warmer clothes are suggested on the next login. These examples may seem futuristic, but such use-cases are feasible in 2016.
In the future, people will generate behavioral data from engaging in ecommerce, and products will generate data that ecommerce companies will want to collect and analyze in order to act on it. Suppliers will generate data. New data types in the future, when combined with available data types now, will enable the creation of data experiences. The combination of customer data, behavioral data, biometric data, mobile data, profile data, customer service data, transaction data, demographic data, and so on opens up avenues of innovation to build consumer experiences based on data.
Data experiences are machine-driven interactions with ecommerce functionality, regardless of device or location, in which the user’s experience is generated or guided from data known about the customer. The simplest example of data experience in 2016 is personalization. Personalization can be driven by recommendations and predictions of what products a known customer may buy next, or the look and feel of a site may be rendered based on preferences. The more an ecommerce site knows about a customer, the richer the types of data experiences can be. To drive these experiences, ecommerce will continue to personalize, but the experiences will be hypercontextual. That means that the ecommerce experiences will appear at the right place, at the right time, with the right offer. Amazon’s Dash Button is a physical example of this Internet of Things–based hypercontextuality that leads to data experiences. Say that you are nearing the end of your bottle of detergent. You push the Dash Button to order the detergent. Doing so may order the product, but it will also give data to Amazon that enables them to understand your purchasing frequency such that predictions could be made about when you might run out of detergent. So in the future Amazon can start to automatically send you detergent delivered just-in-time before you run out. Or your refrigerator detects that your water filter is low, so it prompts you on the touch-interface on the door to reorder. Your coffeemaker and refrigerator share data that estimates when you might run out of coffee, so it gets ordered. Your thermometer detects a rise in your basal temperature, so your mobile phone suggests a remedy. Just-in-time delivery won’t just be made by suppliers; ecommerce sites will use analytics to deliver needed products just-in-time to customers.
These types of data experiences are possible only if many data sources can be stitched together, and only if patterns and behavior can be modeled, predicted, and prescribed and then integrated into business rules and operationalized into ecommerce functionality. This area is rich for innovation. Virtual reality and augmented reality are examples of technologies in which data and analysis renders the experience, and of course commerce will evolve on these platforms. Artificial intelligence will continue to push into ecommerce. Commerce will become conversational, in the sense that you can order products by speaking to a virtual buying assistant. The ecommerce experience could, theoretically, converse with the customer using AI or virtual reality during the browsing, buying, or customer service experiences.
Data experience management will emerge to complement customer experience management. As ecommerce companies create data experiences, analysts will become leaders, managers, and participants on the ecommerce teams that are defining, creating, and ensuring these future capabilities.
New capabilities for analytics tools and technologies will continue to evolve and be released in the future. That’s an easy call to make. What’s harder to identify is what the features and capabilities of these new tools and technologies will be. It’s clear to me that privacy and data security will be a foremost global concern. Data breaches and customer data losses can’t occur, and analytics tools will comply with initiatives and laws in these areas. Business stakeholders and even analytical teams will rely less on information technology professionals for assistance with tools that create data pipelines and data flows to other systems. Enhanced abilities to use natural language, both as a voice-activated method for retrieving data and as a query language, will emerge. Collaboration features to bring together people across geography in both real and virtual environments will grow in importance. The emphasis on predictive will move toward prescriptive recommendations to enable the prediction. Machine learning will continue to be involved in higher-level analysis that touches the business, such that machines will do some of the grunt work of analysts—and eventually artificial intelligence and the analytical automation that will result will change what it means to be an analyst. Let’s dig into some of these future themes here:
• Abstraction of data flows and data pipelines. Currently, the data pipeline from collection to storage to ingestion to curation and governance to cleaning and preparation to analysis, visualization, and data science is fragmented across systems. Although it is unlikely that one tool will be released that does it all, future analytics technology will enable analysts to create a visual abstraction of data flows and the data pipeline. From this abstraction of systems, business rules, analytical methods, and a topology of the objects necessary to create a data flow and analyze the data will exist. Each node in the flow will be an object that can be worked with and defined for use to yield the end result. For example, you might have an object representing a file, another representing the database, others representing data transformations, and others representing applied analytical models through which the data is analyzed. Each object can be explored, configured, and made interoperable and integrated with other objects to weave and stitch together data flows into data pipelines that enable analytics. Objects could be interacted with in augmented or virtual reality.
• Encryption. SSL usage is widespread today for processing ecommerce transactions. Customer data, on the other hand, may not be encrypted, which can represent a security and privacy risk. Analysts in the future will deal with more encrypted data, especially when related to the customer. Even fairly innocuous data like geographic data may be encrypted. Analysts will have to understand the implications of encryption on their work—from a consideration for data availability to the requirement to have a decryption key. As Secure Function Evaluation (SFE) scales and runs faster, it’s conceivable that various types of customer analysis could be supported by SFE.
• Natural language search and conversational speech-enabled features. Querying languages will become simpler and more abstract in the future. The requirement for less technical businesspeople to be able to explore data will create capabilities that allow for the use of natural language and even speech to analyze data. At the simplest, these voice commands will be structured sufficiently to retrieve data, such as “What products are most popular today?” In the future it is entirely possible that speech can be used to direct the type of analysis applied, as in “Run a polynomial regression on data set one.” For now in 2016, the ability to enter queries in natural language to retrieve data and analytics is a step in this direction.
• Automation for self-service and guided-service. Business will continue to demand timely data—that is, data when absolutely needed. The trend for self-service will go beyond giving users the ability to use filters and drill up and down on data. Future tools will take data preferences, business rules, and behavior as input to automate the delivery of reporting and analysis. Business users will specify the data they want to see. The rules for using the data will be understood from an existing semantic layer. Natural language generation technologies will create complex and relevant written reporting in far less time than a human could. Analysts will become less authors of narratives but more of curators and editors of machine-generated narrative analysis.
• Collaboration. Features for collaboration are increasingly more common in analytics tools in 2016. Most typically, these features enable analysts to enter comments and provide qualitative feedback (like star rankings), threaded discussions, and forums. Some tools may enable a person to be designated the business owner of a particular report. New forms of collaboration in analytics tools will include virtual collaboration rooms, including video, audio, and visual immersion into data leading to virtual-reality environments.
• Ease of using predictive analysis. The application of prebuilt models for analyzing data to answer a specific business question will require less data science. Instead of having to solely rely on a data scientist or data science team, analytics models will be prebuilt and then customized to use-cases. Data will be mapped to the models, and the output in its simplest form will be narratives providing predictive results, in formats easily understood by businesspeople. Tools will scan data and use artificial intelligence to determine the best model to apply and may even start testing data in models to refine the predictive power—with the data scientist as a higher-level shepherd of automated processes as opposed to coding them.
• Artificial intelligence and machine learning. These advanced and powerful innovations in computer science will enable prescriptive analytics for ecommerce to become mainstream over the next ten years. Cognitive computing will impact customers, analysts, and employees. Prediction of what might happen will evolve into applications that suggest the best decision to make. Customer experiences will learn customers’ preferences and go beyond personalization to interaction and guided intelligence as part of “data experiences.” You will be able to tell your ecommerce provider to buy you specific items at specific prices, and based on your expressed preferences and some machine-guided interactions, items will be bought for you and sent to you (without your providing a real-time approval or choosing to buy them). The brain of your house will know when something needs to be replaced and will automatically order it for you. Within an ecommerce site, an artificial intelligence will be able to select or identify the best products, promotions, and offers and render them in real time to prospects and customers. Chatbots for ecommerce browsing, purchasing, and servicing will be built on top of AI frameworks.
As you work in ecommerce and use analytics in a noble quest for the truth, you must keep an eye on the future to guide how you evolve today’s innovations into tomorrow’s capabilities. Ecommerce analysis is critical to companies that sell online. It is a capability achieved through more than just tagging a site and looking at the data and reports using one tool and then extracting that data into Excel. The best ecommerce analytics collect raw data, prepare and curate it, understand the data’s profile and lineage, and govern it so it can be analyzed and applied to the highest and best business purpose. Defined data governed by committee, often led by a chief data officer, is managed by line-of-business stewards. Analysts work within an organization structure, which when centralized or constructed as a center of excellence can be led by a chief analytics officer. Controlled data foundations yield usable and accurate data, which can then be ingested in analytical sandboxes where descriptive analysis, reporting, data visualization, and data science can be done. The resulting analysis, prepared automatically through AI and natural language generation or manually, is then socialized through in-person meetings and delivered in online collaborative environments. Raw, curated, or analyzed data may also be streamed into models that automate prediction, guide toward prescriptions, and enable machine-driven, intelligent data experiences. Within the innovative and fascinating current and future ecommerce environment, analysts have an important role to play in creating business value. The data collected, defined, governed, and analyzed by analytics teams is crucial to ecommerce success. Skillful analysis maximizes ecommerce opportunities, creating business value today and promising to create even more future value. Good luck.