17.8 Internet of Things and Dashboards
In the late 1960s, the Internet began as the ARPANET, which initially connected four universities and grew to 10 nodes by the end of 1970.53 In the last 50 years, that has grown to billions of computers, smartphones, tablets and an enormous range of other device types connected to the Internet worldwide. Any device connected to the Internet is a “thing” in the Internet of Things (IoT).
Each device has a unique Internet protocol address (IP address) that identifies it. The explosion of connected devices exhausted the approximately 4.3 billion available IPv4 (Internet Protocol version 4) addresses54 and led to the development of IPv6, which supports approximately 3.4×10 38 addresses (that’s a lot of zeros).55
“Top research firms such as Gartner and McKinsey predict a jump from the 6 billion connected devices we have worldwide today, to 20–30 billion by 2020.”56 Various predictions say that number could be 50 billion. Computer-controlled, Internet-connected devices continue to proliferate. The following is a small subset IoT device types and applications.
| activity trackers—Apple Watch, FitBit, … Amazon Dash ordering buttons Amazon Echo (Alexa), Apple HomePod (Siri), Google Home (Google Assistant) appliances—ovens, coffee makers, refrigerators, … driverless cars earthquake sensors |
healthcare—blood glucose monitors for diabetics, blood pressure monitors, electro-cardiograms (EKG/ECG), electroencephalograms (EEG), heart monitors, ingestible sensors, pacemakers, sleep trackers, … sensors—chemical, gas, GPS, humidity, light, motion, pressure, temperature, … |
smart home—lights, garage openers, video cameras, doorbells, irrigation controllers, security devices, smart locks, smart plugs, smoke detectors, thermostats, air vents tsunami sensors tracking devices wine cellar refrigerators wireless network devices |
IoT Issues
Though there’s a lot of excitement and opportunity in IoT, not everything is positive. There are many security, privacy and ethical concerns. Unsecured IoT devices have been used to perform distributed-denial-of-service (DDOS) attacks on computer systems.57 Home security cameras that you intend to protect your home could potentially be hacked to allow others access to the video stream. Voice-controlled devices are always “listening” to hear their trigger words. This leads to privacy and security concerns. Children have accidentally ordered products on Amazon by talking to Alexa devices, and companies have created TV ads that would activate Google Home devices by speaking their trigger words and causing Google Assistant to read Wikipedia pages about a product to you.58 Some people worry that these devices could be used to eavesdrop. Just recently, a judge ordered Amazon to turn over Alexa recordings for use in a criminal case.59
This Section’s Examples
In this section, we discuss the publish/subscribe model that IoT and other types of applications use to communicate. First, without writing any code, you’ll build a web-based dashboard using Freeboard.io and subscribe to a sample live stream from the PubNub service. Next, you’ll simulate an Internet-connected thermostat which publishes messages to the free Dweet.io service using the Python module Dweepy, then create a dashboard visualization of it with Freeboard.io. Finally, you’ll build a Python client that subscribes to a sample live stream from the PubNub service and dynamically visualizes the stream with Seaborn and a Matplotlib FuncAnimation. In the exercises, you’ll experiment with additional IoT platforms, simulators and live streams.
17.8.1 Publish and Subscribe
IoT devices (and many other types of devices and applications) commonly communicate with one another and with applications via pub/sub (publisher/subscriber) systems. A publisher is any device or application that sends a message to a cloud-based service, which in turn sends that message to all subscribers. Typically each publisher specifies a topic or channel, and each subscriber specifies one or more topics or channels for which they’d like to receive messages. There are many pub/sub systems in use today. In the remainder of this section, we’ll use PubNub and Dweet.io. In the exercises, you can investigate Apache Kafka—a Hadoop ecosystem component that provides a high-performance publish/subscribe service, real-time stream processing and storage of streamed data.
17.8.2 Visualizing a PubNub Sample Live Stream with a Freeboard Dashboard
PubNub is a pub/sub service geared to real-time applications in which any software and device connected to the Internet can communicate via small messages. Some of their common use-cases include IoT, chat, online multiplayer games, social apps and collaborative apps. PubNub provides several live streams for learning purposes, including one that simulates IoT sensors (Section 17.8.5 lists the others).
One common use of live data streams is visualizing them for monitoring purposes. In this section, you’ll connect PubNub’s live simulated sensor stream to a Freeboard.io web-based dashboard. A car’s dashboard visualizes data from your car’s sensors, showing information such as the outside temperature, your speed, engine temperature, the time and the amount of gas remaining. A web-based dashboard does the same thing for data from various sources, including IoT devices.
Freeboard.io is a cloud-based dynamic dashboard visualization tool. You’ll see that, without writing any code, you can easily connect Freeboard.io to various data streams and visualize the data as it arrives. The following dashboard visualizes data from three of the four simulated sensors in the PubNub simulated IoT sensors stream:
For each sensor, we used a Gauge (the semicircular visualizations) and a Sparkline (the jagged lines) to visualize the data. When you complete this section, you’ll see the Gauges and Sparklines frequently moving as new data arrives multiple times per second.
In addition to their paid service, Freeboard.io provides an open-source version (with fewer options) on GitHub. They also provide tutorials that show how to add custom plug-ins, so you can develop your own visualizations to add to their dashboards.
Signing up for Freeboard.io
For this example, register for a Freeboard.io 30-day trial at
https:/Once you’ve registered, the My Freeboards page appears. If you’d like, you can click the Try a Tutorial button and visualize data from your smartphone.
Creating a New Dashboard
In the upper-right corner of the My Freeboards page, enter Sensor Dashboard in the enter a name field, then click the Create New button to create a dashboard. This displays the dashboard designer.
Adding a Data Source
If you add your data source(s) before designing your dashboard, you’ll be able to configure each visualization as you add it:
Under
DATASOURCES, clickADDto specify a new data source.The
DATASOURCEdialog’sTYPEdrop-down list shows the currently supported data sources, though you can develop plug-ins for new data sources as well.60 SelectPubNub. The web page for each PubNub sample live stream specifies theChannelandSubscribe key. Copy these values from PubNub’s Sensor Network page athttps:/, then insert their values in the corresponding/ www.pubnub.com/ developers/ realtime-data-streams/ sensor-network/ DATASOURCEdialog fields. Provide aNAMEfor your data source, then clickSAVE.
Adding a Pane for the Humidity Sensor
A Freeboard.io dashboard is divided into panes that group visualizations. Multiple panes can be dragged to rearrange them. Click the + Add Pane button to add a new pane. Each pane can have a title. To set it, click the wrench icon on the pane, specify Humidity for the TITLE, then click SAVE.
Adding a Gauge to the Humidity Pane
To add visualizations to a pane, click its + button to display the WIDGET dialog. The TYPE drop-down list shows several built-in widgets. Choose Gauge. To the right of the VALUE field, click + DATASOURCE, then select the name of your data source. This displays the available values from that data source. Click humidity to select the humidity sensor’s value. For UNITS, specify %, then click SAVE. This displays the new visualization, which immediately begins showing values from the sensor stream.
Notice that the humidity value has four digits of precision to the right of the decimal point. PubNub supports JavaScript expressions, so you can use them to perform calculations or format data. For example, you can use JavaScript’s function Math.round to round the humidity value to the closest integer. To do so, hover the mouse over the gauge and click its wrench icon. Then, insert "Math.round(" before the text in the VALUE field and ")" after the text, then click SAVE.
Adding a Sparkline to the Humidity Pane
A sparkline is a line graph without axes that’s typically used to give you a sense of how a data value is changing over time. Add a sparkline for the humidity sensor by clicking the humidity pane’s + button, then selecting Sparkline from the TYPE drop-down list. For the VALUE, once again select your data source and humidity, then click SAVE.
Completing the Dashboard
Using the techniques above, add two more panes and drag them to the right of the first. Name them Radiation Level and Ambient Temperature, respectively, and configure each pane with a Gauge and Sparkline as shown above. For the Radiation Level gauge, specify Millirads/Hour for the UNITS and 400 for the MAXIMUM. For the Ambient Temperature gauge, specify Celsius for the UNITS and 50 for the MAXIMUM.
17.8.3 Simulating an Internet-Connected Thermostat in Python
Simulation is one of the most important applications of computers. We used simulation with dice rolling in earlier chapters. With IoT, it’s common to use simulators to test your applications, especially when you do not have access to actual devices and sensors while developing applications. Many cloud vendors have IoT simulation capabilities. In the exercises, you’ll explore the IBM Watson IoT Platform and IOTIFY.io.
Here, you’ll create a script that simulates an Internet-connected thermostat publishing periodic JSON messages—called dweets—to dweet.io. The name “dweet” is based on “tweet”—a dweet is like a tweet from a device. Many of today’s Internet-connected security systems include temperature sensors that can issue low-temperature warnings before pipes freeze or high-temperature warnings to indicate there might be a fire. Our simulated sensor will send dweets containing a location and temperature, as well as low- and high-temperature notifications. These will be True only if the temperature reaches 3 degrees Celsius or 35 degrees Celsius, respectively. In the next section, we’ll use freeboard.io to create a simple dashboard that shows the temperature changes as the messages arrive, as well as warning lights for low- and high-temperature warnings.
Installing Dweepy
To publish messages to dweet.io from Python, first install the Dweepy library:
pip install dweepyThe library is straightforward to use. You can view its documentation at:
https:/Invoking the simulator.py Script
The Python script simulator.py that simulates our thermostat is located in the ch17 example folder’s iot subfolder. You invoke the simulator with two command-line arguments representing the number of total messages to simulate and the delay in seconds between sending dweets:
ipython simulator.py 1000 1Sending Dweets
The simulator.py is shown below. It uses random-number generation and Python techniques that you’ve studied throughout this book, so we’ll focus just on a few lines of code that publish messages to dweet.io via Dweepy. We’ve broken apart the script below for discussion purposes.
By default, dweet.io is a public service, so any app can publish or subscribe to messages. When publishing messages, you’ll want to specify a unique name for your device. We used 'temperature-simulator-deitel-python' (line 17).61 Lines 18–21 define a Python dictionary, which will store the current sensor information. Dweepy will convert this into JSON when it sends the dweet.
1 # simulator.py2 """A connected thermostat simulator that publishes JSON3 messages to dweet.io"""4 import dweepy5 import sys6 import time7 import random89 MIN_CELSIUS_TEMP = -2510 MAX_CELSIUS_TEMP = 4511 MAX_TEMP_CHANGE = 21213 # get the number of messages to simulate and delay between them14 NUMBER_OF_MESSAGES = int(sys.argv[1])15 MESSAGE_DELAY = int(sys.argv[2])1617 dweeter = 'temperature-simulator-deitel-python' # provide a unique name18 thermostat = {'Location': 'Boston, MA, USA',19 'Temperature': 20,20 'LowTempWarning': False,21 'HighTempWarning': False}22
Lines 25–53 produce the number of simulated message you specify. During each iteration of the loop, we
generate a random temperature change in the range –2 to +2 degrees and modify the temperature,
ensure that the temperature remains in the allowed range,
check whether the low- or high-temperature sensor has been triggered and update the thermostat dictionary accordingly,
display how many messages have been generated so far,
use Dweepy to send the message to dweet.io (line 52), and
use the
timemodule’ssleepfunction to wait the specified amount of time before generating another message.23 print('Temperature simulator starting')2425 for message in range(NUMBER_OF_MESSAGES):26 # generate a random number in the range -MAX_TEMP_CHANGE27 # through MAX_TEMP_CHANGE and add it to the current temperature28 thermostat['Temperature'] += random.randrange(29 -MAX_TEMP_CHANGE, MAX_TEMP_CHANGE + 1)3031 # ensure that the temperature stays within range32 if thermostat['Temperature'] < MIN_CELSIUS_TEMP:33 thermostat['Temperature'] = MIN_CELSIUS_TEMP3435 if thermostat['Temperature'] > MAX_CELSIUS_TEMP:36 thermostat['Temperature'] = MAX_CELSIUS_TEMP3738 # check for low temperature warning39 if thermostat['Temperature'] < 3:40 thermostat['LowTempWarning'] = True41 else:42 thermostat['LowTempWarning'] = False4344 # check for high temperature warning45 if thermostat['Temperature'] > 35:46 thermostat['HighTempWarning'] = True47 else:48 thermostat['HighTempWarning'] = False4950 # send the dweet to dweet.io via dweepy51 print(f'Messages sent: {message + 1}\r', end='')52 dweepy.dweet_for(dweeter, thermostat)53 time.sleep(MESSAGE_DELAY)5455 print('Temperature simulator finished')
You do not need to register to use the service. On the first call to dweepy’s dweet_for function to send a dweet (line 52), dweet.io creates the device name. The function receives as arguments the device name (dweeter) and a dictionary representing the message to send (thermostat). Once you execute the script, you can immediately begin tracking the messages on the dweet.io site by going to the following address in your web browser:
https:/If you use a different device name, replace "temperature-simulator-deitel-python" with the name you used. The web page contains two tabs. The Visual tab shows you the individual data items, displaying a sparkline for any numerical values. The Raw tab shows you the actual JSON messages that Dweepy sent to dweet.io.
17.8.4 Creating the Dashboard with Freeboard.io
The sites dweet.io and freeboard.io are run by the same company. In the dweet.io webpage discussed in the preceding section, you can click the Create a Custom Dashboard button to open a new browser tab, with a default dashboard already implemented for the temperature sensor. By default, freeboard.io will configure a data source named Dweet and auto-generate a dashboard containing one pane for each value in the dweet JSON. Within each pane, a text widget will display the corresponding value as the messages arrive.
If you prefer to create your own dashboard, you can use the steps in Section 17.8.2 to create a data source (this time selecting Dweepy) and create new panes and widgets, or you can you modify the auto-generated dashboard.
Below are three screen captures of a dashboard consisting of four widgets:
A
Gaugewidget showing the current temperature. For this widget’sVALUEsetting, we selected the data source’sTemperaturefield. We also set theUNITStoCelsiusand theMINIMUMandMAXIMUMvalues to-25and45degrees, respectively.A
Textwidget to show the current temperature in Fahrenheit. For this widget, we set theINCLUDE SPARKLINEandANIMATE VALUE CHANGEStoYES. For this widget’sVALUEsetting, we again selected the data source’sTemperaturefield, then added to the end of theVALUEfield* 9 / 5 + 32to perform a calculation that converts the Celsius temperature to Fahrenheit. We also specified
Fahrenheitin theUNITSfield.Finally, we added two
Indicator Lightwidgets. For the firstIndicator Light’sVALUEsetting, we selected the data source’sLowTempWarningfield, set theTITLEtoFreezeWarningand set theON TEXTvalue toLOWTEMPERATUREWARNING—ON TEXTindicates the text to display when value istrue. For the secondIndicator Light’sVALUEsetting, we selected the data source’sHighTempWarningfield, set theTITLEtoHigh TemperatureWarningand set theON TEXTvalue toHIGH TEMPERATUREWARNING.
17.8.5 Creating a Python PubNub Subscriber
PubNub provides the pubnub Python module for conveniently performing pub/sub operations. They also provide seven sample streams for you to experiment with—four real-time streams and three simulated streams:62
Twitter Stream—provides up to 50 tweets-per-second from the Twitter live stream and does not require your Twitter credentials.
Hacker News Articles—this site’s recent articles.
State Capital Weather—provides weather data for the U.S. state capitals.
Wikipedia Changes—a stream of Wikipedia edits.
Game State Sync—simulated data from a multiplayer game.
Sensor Network—simulated data from radiation, humidity, temperature and ambient light sensors.
Market Orders—simulated stock orders for five companies.
In this section, you’ll use the pubnub module to subscribe to their simulated Market Orders stream, then visualize the changing stock prices as a Seaborn barplot, like:
Of course, you also can publish messages to streams. For details, see the pubnub module’s documentation at https:/.
To prepare for using PubNub in Python, execute the following command to install the latest version of the pubnub module—the '>=4.1.2' ensures that at a minimum the 4.1.2 version of the pubnub module will be installed:
pip install "pubnub>=4.1.2"The script stocklistener.py that subscribes to the stream and visualizes the stock prices is defined in the ch17 folder’s pubnub subfolder. We break the script into pieces here for discussion purposes.
Message Format
The simulated Market Orders stream returns JSON objects containing five key–value pairs with the keys 'bid_price', 'order_quantity', 'symbol', 'timestamp' and 'trade_type'. For this example, we’ll use only the 'bid_price' and 'symbol'. The PubNub client returns the JSON data to you as a Python dictionary.
Importing the Libraries
Lines 3–13 import the libraries used in this example. We discuss the PubNub types imported in lines 10–13 as we encounter them below.
1 # stocklistener.py2 """Visualizing a PubNub live stream."""3 from matplotlib import animation4 import matplotlib.pyplot as plt5 import pandas as pd6 import random7 import seaborn as sns8 import sys910 from pubnub.callbacks import SubscribeCallback11 from pubnub.enums import PNStatusCategory12 from pubnub.pnconfiguration import PNConfiguration13 from pubnub.pubnub import PubNub14
List and DataFrame Used for Storing Company Names and Prices
The list companies contains the names of the companies reported in the Market Orders stream, and the pandas DataFrame companies_df is where we’ll store each company’s last price. We’ll use this DataFrame with Seaborn to display a bar chart.
15 companies = ['Apple', 'Bespin Gas', 'Elerium', 'Google', 'Linen Cloth']1617 # DataFrame to store last stock prices18 companies_df = pd.DataFrame(19 {'company': companies, 'price' : [0, 0, 0, 0, 0]})20
Class SensorSubscriberCallback
When you subscribe to a PubNub stream, you must add a listener that receives status notifications and messages from the channel. This is similar to the Tweepy listeners you’ve defined previously. To create your listener, you must define a subclass of SubscribeCallback (module pubnub.callbacks), which we discuss after the code:
21 class SensorSubscriberCallback(SubscribeCallback):22 """SensorSubscriberCallback receives messages from PubNub."""23 def __init__(self, df, limit=1000):24 """Create instance variables for tracking number of tweets."""25 self.df = df # DataFrame to store last stock prices26 self.order_count = 027 self.MAX_ORDERS = limit # 1000 by default28 super().__init__() # call superclass's init2930 def status(self, pubnub, status):31 if status.category == PNStatusCategory.PNConnectedCategory:32 print('Connected to PubNub')33 elif status.category == PNStatusCategory.PNAcknowledgmentCategory:34 print('Disconnected from PubNub')3536 def message(self, pubnub, message):37 symbol = message.message['symbol']38 bid_price = message.message['bid_price']39 print(symbol, bid_price)40 self.df.at[companies.index(symbol), 'price'] = bid_price41 self.order_count += 14243 # if MAX_ORDERS is reached, unsubscribe from PubNub channel44 if self.order_count == self.MAX_ORDERS:45 pubnub.unsubscribe_all()46
Class SensorSubscriberCallback’s __init__ method stores the DataFrame in which each new stock price will be placed. The PubNub client calls overridden method status each time a new status message arrives. In this case, we’re checking for the notifications that indicate that we’ve subscribed to or unsubscribed from a channel.
The PubNub client calls overridden method message (lines 36–45) when a new message arrives from the channel. Lines 37 and 38 get the company name and price from the message, which we print so you can see that messages are arriving. Line 40 uses the DataFrame method at to locate the appropriate company’s row and its 'price' column, then assign that element the new price. Once the order_count reaches MAX_ORDERS, line 45 calls the PubNub client’s unsubscribe_all method to unsubscribe from the channel.
Function Update
This example visualizes the stock prices using the animation techniques you learned in Chapter 6’s Intro to Data Science section. Function update specifies how to draw one animation frame and is called repeatedly by the FuncAnimation we’ll define shortly. We use Seaborn function barplot to visualize data from the companies_df DataFrame, using its 'company' column values on the x-axis and 'price' column values on the y-axis.
47 def update(frame_number):48 """Configures bar plot contents for each animation frame."""49 plt.cla() # clear old barplot50 axes = sns.barplot(51 data=companies_df, x='company', y='price', palette='cool')52 axes.set(xlabel='Company', ylabel='Price')53 plt.tight_layout()54
Configuring the Figure
In the main part of the script, we begin by setting the Seaborn plot style and creating the Figure object in which the barplot will be displayed:
55 if __name__ == '__main__':56 sns.set_style('whitegrid') # white background with gray grid lines57 figure = plt.figure('Stock Prices') # Figure for animation58
Configuring the FuncAnimation and Displaying the Window
Next, we set up the FuncAnimation that calls function update, then call Matplotlib’s show method to display the Figure. Normally, this method blocks the script from continuing until you close the Figure. Here, we pass the block=False keyword argument to allow the script to continue so we can configure the PubNub client and subscribe to a channel.
59 # configure and start animation that calls function update60 stock_animation = animation.FuncAnimation(61 figure, update, repeat=False, interval=33)62 plt.show(block=False) # display window63
Configuring the PubNub Client
Next, we configure the PubNub subscription key, which the PubNub client uses in combination with the channel name to subscribe to the channel. The key is specified as an attribute of the PNConfiguration object (module pubnub.pnconfiguration), which line 69 passes to the new PubNub client object (module pubnub.pubnub). Lines 70–72 create the SensorSubscriberCallback object and pass it to the PubNub client’s add_listener method to register it to receive messages from the channel. We use a command-line argument to specify the total number of messages to process.
64 # set up pubnub-market-orders sensor stream key65 config = PNConfiguration()66 config.subscribe_key = 'sub-c-4377ab04-f100-11e3-bffd-02ee2ddab7fe'6768 # create PubNub client and register a SubscribeCallback69 pubnub = PubNub(config)70 pubnub.add_listener(71 SensorSubscriberCallback(df=companies_df,72 limit=int(sys.argv[1] if len(sys.argv) > 1 else 1000))73
Subscribing to the Channel
The following statement completes the subscription process, indicating that we wish to receive messages from the channel named 'pubnub-market-orders'. The execute method starts the stream.
74 # subscribe to pubnub-sensor-network channel and begin streaming75 pubnub.subscribe().channels('pubnub-market-orders').execute()76
Ensuring the Figure Remains on the Screen
The second call to Matplotlib’s show method ensures that the Figure remains on the screen until you close its window.
77 plt.show() # keeps graph on screen until you dismiss its window
Self Check for Section 17.8
(Fill-In) IoT devices (and many other types of devices and applications) commonly communicate with one another and with applications via systems.
Answer: pub/sub (publisher/subscriber)(Fill-In) A(n) is any device or application that sends a message to a cloud-based service, which in turn sends that message to all .
Answer: publisher, subscribers.(Fill-In) A(n) is a graph without axes that’s typically used to give you a sense of how a data value is changing over time.
Answer: sparkline.(Fill-In) In a PubNub Python client that subscribes to a channel, you must create a subclass of , then register an object of that class to receive status notifications and messages from the channel.
Answer:SubscribeCallback.