Linear regression in TensorFlow

In the next examples, we are going to perform linear regression using TensorFlow, helping you understand additional concepts you will need when training and testing your deep learning algorithms.

More specifically, we are going to see three scripts. In each script, we are going to cover the following topics:

Linear regression is a very common statistical method that allows us to model the relationship from a given set of two-dimensional sample points. In this case, the model function is as follows:

This describes a line with the W slope and y-intercept b. Therefore, the goal is to find the values for the W and b parameters that would provide the best fit in some sense (for example, minimizing the mean square error) for the two-dimensional sample points.

When training a linear regression model (see tensorflow_linear_regression_training.py), the first step is to generate some data to use for training the algorithm as follows:

x = np.linspace(0, N, N)
y = 3 * np.linspace(0, N, N) + np.random.uniform(-10, 10, N)

The next step is to define the placeholders in order to feed our training data into the optimizer during the training process as follows:

X = tf.placeholder("float", name='X')
Y = tf.placeholder("float", name='Y')

At this point, we declare two variables (randomly initialized) for the weights and bias as follows:

W = tf.Variable(np.random.randn(), name="W")
b = tf.Variable(np.random.randn(), name="b")

The next step is to construct a linear model as follows:

y_model = tf.add(tf.multiply(X, W), b, name="y_model")

We also define the cost function. In this case, we will use the mean squared error cost function, as demonstrated in the following code snippet:

cost = tf.reduce_sum(tf.pow(y_model - Y, 2)) / (2 * N)

Now, we create the gradient descent optimizer that is going to minimize the cost function modifying the values of the W and b variables.

The traditional optimizer is called gradient descent (iterative optimization algorithm with the aim of finding the minimum of a function), and is shown here:

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

The learning rate parameter controls how much the coefficients change on each update of the gradient descent algorithm. As commented before, the gradient descent is an iterative optimization algorithm and, hence, in each iteration, the parameters are modified according to the learning rate parameter.

The final step when creating the model is to perform the initialization of the variables as follows:

init = tf.global_variables_initializer()

At this point, we can begin the training process inside a session, as demonstrated in the following code snippet:

# Start the training procedure inside a TensorFlow Session:
with tf.Session() as sess:
    # Run the initializer:
    sess.run(init)

    # Uncomment if you want to see the created graph
    # summary_writer = tf.summary.FileWriter(logs_path, sess.graph)

    # Iterate over all defined epochs:
    for epoch in range(training_epochs):

        # Feed each training data point into the optimizer:
        for (_x, _y) in zip(x, y):
            sess.run(optimizer, feed_dict={X: _x, Y: _y})

        # Display the results every 'display_step' epochs:
        if (epoch + 1) % disp_step == 0:
            # Calculate the actual cost, W and b:
            c = sess.run(cost, feed_dict={X: x, Y: y})
            w_est = sess.run(W)
            b_est = sess.run(b)
            print("Epoch", (epoch + 1), ": cost =", c, "W =", w_est, "b =", b_est)

    # Save the final model
    saver.save(sess, './linear_regression')

    # Storing necessary values to be used outside the session
    training_cost = sess.run(cost, feed_dict={X: x, Y: y})
    weight = sess.run(W)
    bias = sess.run(b)

print("Training finished!")

As shown in the preceding code snippet, once the session is started, we run the initializer, and then we iterate over all defined epochs to train the linear regression model. Additionally, we print the results for every display_step epochs. Finally, when the training is done, we save the final model. 

At this point, the training is finished and we can show the results, which can be seen in the following screenshot:

In the preceding diagram, we can see both the training data (left) and the fitted line corresponding to the linear regression model (right).

When saving the final model (saver.save(sess, './linear_regression')), four files are created:

At this point, we can load a pre-trained model and use it for prediction purposes. This is performed in the tensorflow_linear_regression_testing.py script. The first thing to do when loading a model is to load the graph from the .meta file as follows:

tf.reset_default_graph()
imported_meta = tf.train.import_meta_graph("linear_regression.meta")

The second step is to load the values of the variables (note that values only exist within a session). We also run the model to get the values of W, b, and new prediction values as follows:

with tf.Session() as sess:
    imported_meta.restore(sess, './linear_regression')
    # Run the model to get the values of the variables W, b and new prediction values:
    W_estimated = sess.run('W:0')
    b_estimated = sess.run('b:0')
    new_predictions = sess.run(['y_model:0'], {'X:0': new_x})

At this point, we can show the training data, the regression line, and the newly obtained predictions, which can be seen in the following screenshot:

As shown in the previous screenshot, we have used the pre-trained model for making new predictions (blue points). However, when serving a model in production, we just want our model and its weights nicely packaged in one file, facilitating storage, versioning, and updates of your different models. The result will be a binary file with a .pb extension, containing both topology and weights of the trained network. The process of creating this binary file, and how to use it for inference, is performed in the tensorflow_save_and_load_using_model_builder.py script.

In this script, we have coded the export_model() function to export the trained model using SaveModel (https://www.tensorflow.org/guide/saved_model) as follows:

def export_model():
    """Exports the model"""

    trained_checkpoint_prefix = 'linear_regression'

    loaded_graph = tf.Graph()
    with tf.Session(graph=loaded_graph) as sess:
        sess.run(tf.global_variables_initializer())

        # Restore from checkpoint
        loader = tf.train.import_meta_graph(trained_checkpoint_prefix + '.meta')
        loader.restore(sess, trained_checkpoint_prefix)

        # Add signature:
        ...
        signature_map = {signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature}

        # Export model:
        builder = tf.saved_model.builder.SavedModelBuilder('./my_model')
        builder.add_meta_graph_and_variables(sess, signature_def_map=signature_map,
                                             tags=[tf.saved_model.tag_constants.SERVING])
        builder.save()

This will create saved_model.pb inside the my_model folder. At this point, in order to verify whether the exported model has been generated correctly, we can both import and use it in order to make new predictions as follows:

with tf.Session(graph=tf.Graph()) as sess:
    tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], './my_model')
    graph = tf.get_default_graph()
    x = graph.get_tensor_by_name('X:0')
    model = graph.get_tensor_by_name('y_model:0')
    print(sess.run(model, {x: new_x}))

After calling the load function, the graph will be loaded as the default graph. Additionally, the variables are also loaded, so you can start running inference on any new data. This will output the [153.04472 166.54755 180.05037] array, which corresponds to the predicted values generated by our model.