Log In
Or create an account ->
Imperial Library
Home
About
News
Upload
Forum
Help
Login/SignUp
Index
Title Page
Copyright
Stream Analytics with Microsoft Azure
Credits
About the Authors
About the Reviewers
www.PacktPub.com
Why subscribe?
Customer Feedback
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
Introducing Stream Processing and Real-Time Insights
Understanding stream processing
Understanding queues, Pub/Sub, and events
Queues
Publish and Subscribe model
Real-world implementations of the Publish/Subscribe model
Azure implementation of queues and Publish/Subscribe models
What is an event?
Event streaming
Event correlation
Azure implementation of event processing
Architectural components of Event Hubs
Simple event processing
Event stream processing
Complex event processing
Summary
Introducing Azure Stream Analytics and Key Advantages
Services offered by Microsoft
Introduction to Azure Stream Analytics
Configuration of Azure Stream Analytics
Key advantages of Azure Stream Analytics
Security
Programmer productivity
Declarative SQL constructs
Built-in temporal semantics
Lowest total cost of ownership
Mission-critical and enterprise-less scalability and availability
Global compliance
Microsoft Cortana Intelligence suite integration
Azure IoT integration
Summary
Designing Real-Time Streaming Pipelines
Differencing stream processing and batch processing
Logical flow of processing
Out of order and late arrival of data
Session grouping and windowing challenges
Message consistency
Fault tolerance, recovery, and storage
Source
Communication and collection
Ingest, queue, and transform
Hot path
Cold path
Data retention
Presentation and action
Canonical Azure architecture
Summary
Developing Real-Time Event Processing with Azure Streaming
Stream Analytics tools for Visual Studio
Prerequisites for the installation of Stream Analytics tools
Development of a Stream Analytics job using Visual Studio
Defining a Stream Analytics query for Vehicle Telemetry job analysis using Stream Analytics tools
Query to define Vehicle Telemetry (Connected Car) engine health status and pollution index over cities
Testing Stream Analytics queries locally or in the cloud
Stream Analytics job configuration parameter settings in Visual Studio
Implementation of an Azure Stream Analytics job using the Azure portal
Provisioning for an Azure Stream Analytics job using the Azure Resource Manager template
Azure ARM Template - Infrastructure as code
Getting started with provisioning Azure Stream Analytics job using the ARM template
Deployment and validation of the Stream Analytics ARM template to Azure Resource Group
Configuration of the Azure Streaming job with different input data sources and output data sinks
Data input types-data stream and reference data
Data Stream inputs
Reference data
Job topology output data sinks of Stream Analytics
Summary
Building Using Stream Analytics Query Language
Built-in functions
Scalar functions
Aggregate and analytic functions
Array functions
Other functions
Data types and formats
Complex types
Query language elements
Windowing
Tumbling windows
Hopping windows
Sliding windows
Time management and event delivery guarantees
Summary
How to achieve Seamless Scalability with Automation
Understanding parts of a Stream Analytics job definition (input, output, reference data, and job)
Deployment of Azure Stream Analytics using ARM template
Configuring input
Configuring output
Building the sample test code
How to scale queries using Streaming units and partitions
Application and Arrival Time
Partitions
Input source
Output source
Embarrassingly parallel jobs and Not embarrassingly parallel jobs
Sample use case
Configuring SU using Azure portal
Out of order and late-arriving events
Summary
Integration of Microsoft Business Intelligence and Big Data
What is Big Data Lambda Architecture?
Concepts of batch processing and stream processing in data analytics
Specifications for slow/cold path of data - batch data processing
Moving to the streaming-based data solution pattern
Evolution of Kappa Architecture and benefits
Comparison between Azure Stream Analytics and Azure HDInsight Storm
Designing data processing pipeline of an interactive visual dashboard through Stream Analytics and Power BI
Integrating Power BI as an output job connector for Stream Analytics
Summary
Designing and Managing Stream Analytics Jobs
Reference data streams with Azure Stream Analytics
Configuration of Reference data for Azure Stream Analytics jobs
Integrating a reference data stream as job topology input for an Azure Stream Analytics job
Stream Analytics query configuration for Reference Data join
Refresh schedule of a reference data stream
Configuration of output data sinks for Azure Stream Analytics with Azure Data Lake Store
Configuring Azure Data Lake Store as an output data sink of Stream Analytics
Configuring Azure Data Lake Store as an output sink of Stream Analytics jobs
Configuring Azure Cosmos DB as an output data sink for Azure Stream Analytics
Features of Azure Cosmos DB for configuring output sinks of Azure Stream Analytics
Configuring Azure Cosmos DB integrated with Azure Stream Analytics as an output sink
Stream Analytics job output to Azure Function Apps as Serverless Architecture
Provisioning steps to an Azure Function
Configuring an Azure function as a serverless architecture model integrated with Stream Analytics job output
Summary
Optimizing Intelligence in Azure Streaming
Integration of JavaScript user-defined functions using Azure Stream Analytics
Adding JavaScript UDF with a Stream Analytics job
Stream Analytics and JavaScript data type conversions
Integrating intelligent Azure machine learning algorithms with Stream Analytics function
Data pipeline Streaming application building concepts using Azure .NET Management SDK
Implementation steps of Azure Stream Analytics jobs using .NET management SDK
Summary
Understanding Stream Analytics Job Monitoring
Troubleshooting with job metrics
Visual monitoring of job diagram
Logging of diagnostics logs
Enabling diagnostics logs
Exploring the logs sent to the storage account
Configuring job alerts
Viewing resource health information with Azure resource health
Exploring different monitoring experiences
Building a monitoring dashboard
Summary
Use Cases for Real-World Data Streaming Architectures
Solution architecture design and Proof-of-Concept implementation of social media sentiment analytics using Twitter and a sentiment analytics dashboard
Definition of sentiment analytics
Prerequisites required for the implementation of Twitter sentiment analytics PoC
Steps for implementation of Twitter sentiment analytics
Remote monitoring analytics using Azure IoT Suite
Provisioning of remote device monitoring analytics using Azure IoT Suite
Implementation of a connected factory use case using Azure IoT Suite
Connected factory solution with Azure IoT Suite
Real-world telecom fraud detection analytics using Azure Stream Analytics and Cortana Intelligence Gallery with interactive visuals from Microsoft Power BI
Implementation steps of fraud detection analytics using Azure Stream Analytics
Steps for building the fraud detection analytics solution
Summary
← Prev
Back
Next →
← Prev
Back
Next →