Preparing the data

The moment you get your first look at the data, we will move on to preparing MXNet arrays and creating data providers. The process is very similar to the one we had when working with MNIST except that now we need to handle a dataset of a higher dimensionality. This time, we will also include the creation of a data provider for test images.

Let's start by reserving the memory for our train, validation, and test datasets. We will be creating 4 dimensional MXNet arrays to store the future values:

train_length = 40000
validation_length = 10000

train_data_array = mx.zeros((size(train_x, 1, 2, 3)..., train_length...));
train_label_array = mx.zeros(train_length);

validation_data_array = mx.zeros((size(train_x, 1, 2, 3)..., validation_length...));
validation_label_array = mx.zeros(validation_length);

test_data_array = mx.zeros((size(train_x, 1, 2, 3)..., size(test_x, 4)...));
test_label_array = mx.zeros(size(test_x, 4));

Now, we will iterate over the datasets and populate the arrays:

# The number of records we send to the training should be at least number of outcome
for idx = 1:train_length
train_data_array[idx:idx] = reshape(train_x[:, :, :, idx], (size(train_x, 1, 2,3 )..., 1...))
train_label_array[idx:idx] = train_y[idx]
end

for idx = 1:validation_length
validation_data_array[idx:idx] = reshape(train_x[:, :, :, train_length + idx], (size(train_x, 1, 2, 3)..., 1...))
validation_label_array[idx:idx] = train_y[train_length + idx]
end

for idx = 1:size(test_x, 4)
test_data_array[idx:idx] = reshape(test_x[:, :, :, idx], (size(test_x, 1, 2,3 )..., 1...))
test_label_array[idx:idx] = test_y[idx]
end

The moment the datasets are populated, we convert and connect the image data and classes using DataProvider:

train_data_provider = mx.ArrayDataProvider(:data => train_data_array, :label => train_label_array, batch_size = 100, shuffle = true);
validation_data_provider = mx.ArrayDataProvider(:data => validation_data_array, :label => validation_label_array, batch_size = 100, shuffle = true);
test_data_provider = mx.ArrayDataProvider(:data => test_data_array, :label => test_label_array, batch_size = 100);

We can now move on and proceed with building and testing the neural network.