Creating an image data iterator

In the two previous tasks, images were available directly from the MLDatasets package. Unfortunately, in real-world scenarios, images are stored on a disk. 

Despite the large number of records in the training dataset, we will put them all into memory. To make the process more efficient, we will be resizing the images to 32x32 pixels and converting them to floats. This is shown in the following code:

files = readdir(IMAGES_PATH);
data_x = zeros((32, 32, 3, size(files, 1)));
data_y = zeros(size(files, 1));

for idx = 1:size(files, 1)
    file_name = joinpath(IMAGES_PATH, files[idx])

    if endswith(file_name, ".jpg")
        img = imresize(load(file_name), (32, 32))
        data_x[:, :, :, idx] = permuteddimsview(Float16.(channelview(img)), (2, 3, 1))
        data_y[idx] = 1 * contains(files[idx], "dog")
    end
end

The following two class labels are assigned:

• 0 – cat
• 1 – dog

Because of the way we are processing files in the directory, dogs will follow after cats. It makes the dataset ordered and requires shuffling prior to building the data providers. We will be using an 80/20 split. 80% of the data will be used for training and the rest will be used for validation. This is shown in the following code:

total_count = size(data_y, 1);
indices = shuffle(1:total_count);
train_indices = indices[1:Int(total_count * 0.8)];
validation_indices = indices[Int(total_count * 0.8) + 1:total_count];

Now, we are ready to create the data provider and reference the indices that were defined in the preceding code:

train_data_provider = mx.ArrayDataProvider(:data => data_x[:, :, :, train_indices], :label => data_y[train_indices], batch_size = 100, shuffle = true);
validation_data_provider = mx.ArrayDataProvider(:data => data_x[:, :, :, validation_indices], :label => data_y[validation_indices], batch_size = 100, shuffle = true);

We are all set for model training!