Using TensorBoard to visualize deep learning networks

Computation graphs in TensorFlow can be very complex, so there is a visualization tool called TensorBoard to visualize these graphs and assist in debugging. TensorBoard can plot a computation graph, display metrics from training, and so on. Since Keras uses TensorFlow in the backend, it too can use TensorBoard. Here is the MNIST example from Keras with TensorBoard logging enabled. This code can be found in the Chapter8/mnist_keras.R folder. The first part of the code loads the data, pre-processes it, and defines the model architecture. Hopefully, this should be familiar to you at this stage:

library(keras)

mnist_data <- dataset_mnist()
xtrain <- array_reshape(mnist_data$train$x,c(nrow(mnist_data$train$x),28,28,1))
ytrain <- to_categorical(mnist_data$train$y,10)
xtrain <- xtrain / 255.0

model <- keras_model_sequential()
model %>%
  layer_conv_2d(filters=32,kernel_size=c(5,5),activation='relu',
                input_shape=c(28,28,1)) %>% 
  layer_max_pooling_2d(pool_size=c(2,2)) %>% 
  layer_dropout(rate=0.25) %>% 
  layer_conv_2d(filters=32,kernel_size=c(5,5),activation='relu') %>% 
  layer_max_pooling_2d(pool_size=c(2,2)) %>% 
  layer_dropout(rate=0.25) %>% 
  layer_flatten() %>% 
  layer_dense(units=256,activation='relu') %>% 
  layer_dropout(rate=0.4) %>% 
  layer_dense(units=10,activation='softmax')

model %>% compile(
  loss=loss_categorical_crossentropy,
  optimizer="rmsprop",metrics="accuracy"
)

To enable logging, add a callbacks parameter to the model.fit function to tell Keras/TensorFlow to log events to a directory. The following code will output log data to the /tensorflow_logs directory:

model %>% fit(
  xtrain,ytrain,
  batch_size=128,epochs=10,
  callbacks=callback_tensorboard("/tensorflow_logs",
                                 histogram_freq=1,write_images=0),
  validation_split=0.2
)
# from cmd line,run 'tensorboard --logdir /tensorflow_logs'

Warning: The event logs can take up a lot of space. For 5 epochs on the MNIST dataset, 1.75 GB of information was created. Most of this was because of the image data that was included, so you may consider setting write_images=0 to reduce the size of the logs.

TensorBoard is a web application, and you must start the TensorBoard program for it to run. When the model has finished training, follow these steps to start the TensorBoard web application:

1. Open up Command Prompt and enter the following:

$ tensorboard --logdir /tensorflow_logs

2. If TensorBoard starts successfully, you should get a message similar to the following at Command Prompt:

TensorBoard 0.4.0rc2 at http://xxxxxx:6006 (Press CTRL+C to quit)

3. Open a web browser to the link that was provided. The web page should be similar to the following:



Figure 8.1: TensorBoard – model metrics
4. The preceding screenshot shows us the model metrics on the training and validation test sets – these are similar to the metrics shown in RStudio during training:



Figure 8.2: RStudio – model metrics
5. If you click on the Images option, you will be able to visualize the layers in the model and see how they change over epochs:



Figure 8.3: TensorBoard – visualizing the model layers
6. If you click on the Graphs option, it will show the computation graph for the model. You can also download it as an image file. Here is the computation graph for this model:



Figure 8.4: TensorBoard – computation graph

Some of this will seem familiar. We can see our convolutional, max pooling, flatten, dense, and dropout layers. The rest are not as obvious. As a higher-level abstraction, Keras takes care of a lot of the complexities in creating the computation graph.

7. By clicking on the Histogram option, you can see how the distribution of tensors changes over time:



Figure 8.5: TensorBoard – histograms

It is possible to use TensorBoard to debug a model. For example, it would be possible to investigate a vanishing gradient or an exploding gradient problem to see where the weights of the model were either vanishing to zero or exploding to infinity. There is a lot more to TensorBoard, so if you are curious you can follow the online documentation on it.

In the next section, we will use TensorFlow to build a regression model and a convolutional neural network.