Orhan Gazi Yalçın

Applied Neural Networks with TensorFlow 2

API Oriented Deep Learning with Python

1st ed.
Orhan Gazi Yalçın
Istanbul, Turkey
ISBN 978-1-4842-6512-3e-ISBN 978-1-4842-6513-0
https://doi.org/10.1007/978-1-4842-6513-0
Apress Standard
© Orhan Gazi Yalçın 2021
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Distributed to the book trade worldwide by Springer Science+Business Media New York,1 NY Plazar, New York, NY 10014. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.

I dedicate this book to my overcurious dad, Lutfi – who kept sneaking into the study room to see how far I was into the book – and to my mom, Ayşe, for always supporting and encouraging me.

I would also like to thank my friend, Enes, for encouraging me to write this book in the first place.

Finally, I would like to thank my sister and brother, Merve and Kürşat, and all my friends who supported me throughout the whole process – all the way – till the last word.

../images/501289_1_En_BookFrontmatter_Figb_HTML.jpg
Acknowledgments

This book was written during a global lockdown due to the Covid-19 pandemic, which created a new normal that I have never experienced before. Writing a book in the middle of a global crisis was a very intense experience, and I was uncertain about taking this responsibility for a long time. Thanks to my family and friends, I was able to complete the book even earlier than scheduled. Now I am glad that I accepted Aaron’s invitation, who guided me throughout the whole process. Thank you very much for reaching out to me in the first place and making it possible to have this book written.

I would like to thank Jessica Vakili for coordinating the entire project and for being there whenever I needed. I would also like to thank Vishwesh Ravi Shrimali for reviewing every single line of the book and providing me with all the valuable comments, which helped to improve the quality of the book tremendously.

Being surrounded with people who all have a positive attitude made this experience very fruitful, and I am looking forward to working with them in the future. Thank you all very much!

../images/501289_1_En_BookFrontmatter_Figd_HTML.jpg
Table of Contents
Chapter 1:​ Introduction 1
Python as Programming Language 3
Timeline of Python 3
Python 2 vs.​ Python 3 4
Why Python?​ 5
TensorFlow As Deep Learning Framework 7
Timeline of TensorFlow 8
Why TensorFlow?​ 10
What’s New in TensorFlow 2.​x 10
TensorFlow Competitors 15
Installation and Environment Setup 20
Interactive Programming Environments:​ IPython, Jupyter Notebook, and Google Colab 21
IPython 22
Jupyter Notebook 23
Google Colab 30
Hardware Options and Requirements 32
Chapter 2:​ Introduction to Machine Learning 33
What Is Machine Learning?​ 33
Scope of Machine Learning and Its Relation to Adjacent Fields 37
Artificial Intelligence 37
Deep Learning 38
Data Science 39
Big Data 39
The Taxonomy Diagram 39
Machine Learning Approaches and Models 40
Supervised Learning 41
Unsupervised Learning 44
Semi-supervised Learning 46
Reinforcement Learning 47
Evaluation of Different Approaches 48
Steps of Machine Learning 49
Gathering Data 50
Preparing Data 50
Model Selection 51
Training 51
Evaluation 52
Hyperparameter Tuning 54
Prediction 55
Final Evaluations 55
Chapter 3:​ Deep Learning and Neural Networks Overview 57
Timeline of Neural Networks and Deep Learning Studies 59
Structure of Artificial Neural Networks 63
McCulloch-Pitts Neuron 63
Linear Threshold Unit (LTU) 64
Perceptron 64
A Modern Deep Neural Network 65
Activation Functions 66
Loss (Cost or Error) Functions 69
Optimization in Deep Learning 70
Backpropagation 71
Optimization Algorithms 72
Optimization Challenges 75
Overfitting and Regularization 77
Overfitting 77
Regularization 78
Feature Scaling 79
Final Evaluations 80
Chapter 4:​ Complementary Libraries to TensorFlow 2.​x 81
Installation with Pip 82
NumPy – Array Processing 85
SciPy – Scientific Computing 86
Pandas – Array Processing and Data Analysis 88
Matplotlib and Seaborn – Data Visualization 89
Scikit-learn – Machine Learning 91
Flask – Deployment 93
Final Evaluations 94
Chapter 5:​ A Guide to TensorFlow 2.​0 and Deep Learning Pipeline 95
TensorFlow Basics 96
Eager Execution 96
Tensor 97
Variable 99
TensorFlow Deep Learning Pipeline 100
Data Loading and Preparation 101
Dataset Object (tf.​data.​Dataset) 101
TensorFlow Datasets Catalog 102
NumPy Array 106
Pandas DataFrame 106
Other Objects 107
Model Building 107
Keras API 108
Estimator API 112
Compiling, Training, and Evaluating the Model and Making Predictions 113
The Standard Method 113
Custom Training 116
Saving and Loading the Model 118
Saving the Model 119
Loading the Model 120
Conclusion 120
Chapter 6:​ Feedforward Neural Networks 121
Deep and Shallow Feedforward Neural Networks 122
Shallow Feedforward Neural Network 122
Deep Feedforward Neural Network 123
Feedforward Neural Network Architecture 125
Layers in a Feedforward Neural Network 125
Input Layer 125
Output Layer 126
Hidden Layer 126
Case Study | Fuel Economics with Auto MPG 127
Initial Installs and Imports 127
Downloading the Auto MPG Data 128
Data Preparation 129
Model Building and Training 134
Evaluating the Results 138
Making Predictions with a New Observation 141
Conclusion 143
Chapter 7:​ Convolutional Neural Networks 145
Why Convolutional Neural Networks?​ 146
CNN Architecture 147
Layers in a CNN 147
A Full CNN Model 151
Case Study | Image Classification with MNIST 152
Downloading the MNIST Data 152
Reshaping and Normalizing the Images 154
Building the Convolutional Neural Network 155
Compiling and Fitting the Model 156
Evaluating the Model 157
Saving the Trained Model 159
Conclusion 160
Chapter 8:​ Recurrent Neural Networks 161
Sequence Data and Time-Series Data 161
RNNs and Sequential Data 163
The Basics of RNNs 164
The History of RNNs 164
Applications of RNNs 165
Mechanism of RNNs 166
RNN Types 167
Simple RNNs 168
Long Short-Term Memory (LSTM) 169
Gated Recurrent Units (GRUs) 170
Case Study | Sentiment Analysis with IMDB Reviews 171
Preparing Our Colab for GPU Accelerated Training 172
IMDB Reviews 173
Preparing the Dataset 175
Building the Recurrent Neural Network 176
Compiling and Fitting the Model 178
Evaluating the Model 179
Making New Predictions 181
Saving and Loading the Model 182
Conclusion 185
Chapter 9:​ Natural Language Processing 187
History of NLP 187
Early Ideas 188
Rule-Based NLP 188
Statistical NLP and Supervised Learning 189
Unsupervised and Semi-supervised NLP 190
Real-World Applications of NLP 190
Major Evaluations, Techniques, Methods, and Tasks 191
Morphosyntax 192
Semantics 193
Discourse 195
Speech 195
Dialogue 196
Cognition 196
Natural Language Toolkit (NLTK) 196
Case Study | Text Generation with Deep NLP 198
The Goal of the Case Study 198
Shakespeare Corpus 199
Initial Imports 200
Loading the Corpus 201
Vectorize the Text 202
Creating the Dataset 203
Building the Model 205
Compiling and Training the Model 207
Generating Text with the Trained Model 209
Conclusion 213
Chapter 10:​ Recommender Systems 215
Popular Approaches 216
Collaborative Filtering 216
Data Collection 217
Content-Based Filtering (Personality-Based Approach) 219
Other Recommender System Approaches 220
Case Study | Deep Collaborative Filtering with MovieLens Dataset 221
MovieLens Dataset 222
Initial Imports 222
Loading the Data 223
Processing the Data 225
Splitting the Dataset 227
Building the Model 228
Compile and Train the Model 231
Make Recommendations 232
Conclusion 236
Chapter 11:​ Autoencoders 237
Advantages and Disadvantages of Autoencoders 238
Autoencoder Architecture 239
Layers Used in an Autoencoder 240
Advantages of Depth 241
Variations of Autoencoders 241
Undercomplete Autoencoders 242
Regularized Autoencoders 242
Variational Autoencoder (VAE) 244
Use Cases of Autoencoders 245
Case Study | Image Denoising with Fashion MNIST 246
Fashion MNIST Dataset 247
Initial Imports 247
Loading and Processing the Data 248
Adding Noise to Images 251
Building the Model 253
Denoising Noisy Images 255
Conclusion 257
Chapter 12:​ Generative Adversarial Network 259
Method 259
Architecture 260
GAN Components 261
A Known Issue:​ Mode Collapse 262
Final Notes on Architecture 262
Applications of GANs 263
Art and Fashion 263
Manufacturing, Research, and R&​D 263
Video Games 264
Malicious Applications and Deep Fake 264
Miscellaneous Applications 264
Case Study | Digit Generation with MNIST 265
Initial Imports 265
Load and Process the MNIST Dataset 266
Build the GAN Model 267
Train the GAN Model 275
Animate Generated Digits During the Training 281
Conclusion 284
Index 285
About the Author
Orhan Gazi Yalçın
../images/501289_1_En_BookFrontmatter_Figc_HTML.jpg
is a joint PhD candidate at the University of Bologna and the Polytechnic University of Madrid. After completing his double major in business and law, he began his career in Istanbul, working for a city law firm, Allen & Overy, and a global entrepreneurship network, Endeavor. During his academic and professional career, he taught himself programming and excelled in machine learning. He currently conducts research on hotly debated law and AI topics such as explainable artificial intelligence and the right to explanation by combining his technical and legal skills. In his spare time, he enjoys free diving, swimming, exercising, as well as discovering new countries, cultures, and cuisines.
 
About the Technical Reviewer
Vishwesh Ravi Shrimali

graduated from BITS Pilani in 2018, where he studied mechanical engineering. Since then, he has worked with BigVision LLC on deep learning and computer vision and was involved in creating official OpenCV AI courses. Currently, he is working at Mercedes Benz Research and Development India Pvt. Ltd. He has a keen interest in programming and AI and has applied that interest in mechanical engineering projects. He has also written multiple blogs on OpenCV and deep learning on LearnOpenCV, a leading blog on computer vision. He has also coauthored Machine Learning for OpenCV4 (second edition) by Packt. When he is not writing blogs or working on projects, he likes to go on long walks or play his acoustic guitar.