17 Big Data: Hadoop, Spark, NoSQL and IoT

Objectives

In this chapter you’ll:

• Understand what big data is and how quickly it’s getting bigger.

• Manipulate a SQLite relational database using Structured Query Language (SQL).

• Understand the four major types of NoSQL databases.

• Store tweets in a MongoDB NoSQL JSON document database and visualize them on a Folium map.

• Understand Apache Hadoop and how it’s used in big-data batch-processing applications.

• Build a Hadoop MapReduce application on Microsoft’s Azure HDInsight cloud service.

• Understand Apache Spark and how it’s used in high-performance, real-time big-data applications.

• Use Spark streaming to process data in mini-batches.

• Understand the Internet of Things (IoT) and the publish/subscribe model.

• Publish messages from a simulated Internet-connected device and visualize its messages in a dashboard.

• Subscribe to PubNub’s live Twitter and IoT streams and visualize the data.

Outline

1. 17.1 Introduction

2. 17.2 Relational Databases and Structured Query Language (SQL)

   1. 17.3.1 A books Database

   2. 17.3.2 SELECT Queries

   3. 17.3.3 WHERE Clause

   4. 17.3.4 ORDER BY Clause

   5. 17.3.5 Merging Data from Multiple Tables: INNER JOIN

   6. 17.3.6 INSERT INTO Statement

   7. 17.3.7 UPDATE Statement

   8. 17.3.8 DELETE FROM Statement

3. 17.3 NoSQL and NewSQL Big-Data Databases: A Brief Tour

   1. 17.3.1 NoSQL Key–Value Databases

   2. 17.3.2 NoSQL Document Databases

   3. 17.3.3 NoSQL Columnar Databases

   4. 17.3.4 NoSQL Graph Databases

   5. 17.3.5 NewSQL Databases

4. 17.4 Case Study: A MongoDB JSON Document Database

   1. 17.4.1 Creating the MongoDB Atlas Cluster

   2. 17.4.2 Streaming Tweets into MongoDB

5. 17.5 Hadoop

   1. 17.5.1 Hadoop Overview

   2. 17.6.2 Summarizing Word Lengths in Romeo and Juliet via MapReduce

   3. 17.5.3 Creating an Apache Hadoop Cluster in Microsoft Azure HDInsight

   4. 17.5.4 Hadoop Streaming

   5. 17.5.5 Implementing the Mapper

   6. 17.5.6 Implementing the Reducer

   7. 17.5.7 Preparing to Run the MapReduce Example

   8. 17.5.8 Running the MapReduce Job

6. 17.6 Spark

   1. 17.6.1 Spark Overview

   2. 17.6.2 Docker and the Jupyter Docker Stacks

   3. 17.6.3 Word Count with Spark

   4. 17.6.4 Spark Word Count on Microsoft Azure

7. 17.7 Spark Streaming: Counting Twitter Hashtags Using the pyspark-notebook Docker Stack

   1. 17.7.1 Streaming Tweets to a Socket

   2. 17.7.2 Summarizing Tweet Hashtags; Introducing Spark SQL

8. 17.8 Internet of Things and Dashboards

   1. 17.8.1 Publish and Subscribe

   2. 17.8.2 Visualizing a PubNub Sample Live Stream with a Freeboard Dashboard

   3. 17.8.3 Simulating an Internet-Connected Thermostat in Python

   4. 17.8.4 Creating the Dashboard with Freeboard.io

   5. 17.8.5 Creating a Python PubNub Subscriber

9. 17.9 Wrap-Up

1. Exercises