Training and validating the results

The next step will be to prepare a training and validation dataset. Our dataset has over 9,000 images and I have made a decision to run the training on the 8000 and leave the rest for validation. I have also defined batch_size for training to be as large as the validation dataset. You might need to adjust it to 100, 250, or 500 if you don't have enough resources:

train_indices = 1:8000;
valid_indices = 8000:length(images);
batch_size = length(valid_indices);

Next, we will create a custom iterator. It will be a for loop that will randomly choose images for training and validate them on our validation dataset:

for i = 1:30

    println(i, " - ", i+batch_size-1);

    train_indices_batch = sample(train_indices, batch_size, replace = false);
    valid_indices_batch = sample(valid_indices, batch_size, replace = false);

    train_data_provider = mx.ArrayDataProvider(:data => features[:, train_indices_batch], :label => map(x -> x[2], images[train_indices_batch]), batch_size = batch_size);
    validation_data_provider = mx.ArrayDataProvider(:data => features[:, valid_indices_batch], :label => map(x -> x[2], images[valid_indices_batch]), batch_size = batch_size);

    mx.fit(nnet_new, optimizer, train_data_provider, eval_data = validation_data_provider, n_epoch = 1, callbacks = [mx.speedometer()]);

end

We can infer the following from the preceding code—I have used the sample function from the StatsBase package. It allows me to select indices without replacement. I have referenced both the features and images datasets. The features dataset stores features generated by Inception V3 and the images dataset stores the class values.

Depending on your batch_size, it can take up to 30 iterations to reach 90%. This 90% is considered to be a very good result. You won't be able to reach it by using the classic computer vision technique:

INFO: Start training on MXNet.mx.Context[GPU0]
INFO: Initializing parameters...
INFO: Creating KVStore...
INFO: TempSpace: Total 9 MB allocated on GPU0
INFO: Start training...
INFO: == Epoch 001/001 ==========
INFO: ## Training summary
INFO: accuracy = 0.9144
INFO: time = 0.0041 seconds
INFO: ## Validation summary
INFO: accuracy = 0.9100
INFO: Finish training on MXNet.mx.Context[GPU0]