When we dealt with smaller datasets, it was enough for us to just load the entire dataset into computer memory. This is simple and works fine if your dataset is small enough; however, a lot of the time, this won't be the case. We will now look at how to overcome this issue.

In order to avoid loading all our data at once, we will need to create a data pipeline that can feed our training data to the model. This pipeline will be responsible for, among other things, loading a batch of elements from storage, preprocessing the data, and finally, feeding the data to our model. Luckily for us, this can all be easily accomplished using the TensorFlow data API.

For these examples, we are going to assume that we have saved our data into multiple (two in this case) TFRecord files like those described previously. There is no difference if you have more than two; you just have to include all their names when setting things up.

We start by creating a TFRecord dataset from a list of all the TFRecord file names:

 # Create a TFRecord dataset that reads all of the Examples from  
   two files.
 train_filenames= ["/data/train1.tfrecord", "/data/train2.tfrecord"]
 train_dataset = tf.data.TFRecordDataset(filenames)

Next, we have to decode our TFRecords. To do this, we write a function that will take in a TFRecord, decode it and then return an input image and its corresponding label:

# Function for decoding our TFRecord. We assume our images are fixed size 224x224x3
 def decode_tfrec(proto_in):

    my_features = {'image_raw': tf.FixedLenFeature([], tf.string),
                'Label': tf.FixedLenFeature([], tf.int64)}
    parsed_features = tf.parse_single_example(proto_in, features=my_features)image = tf.decode_raw(parsed_features['image_raw'], tf.uint8)
    image = tf.cast(image, tf.float32)  # Tensorflow data needs to be float32.
    image = tf.reshape(image, [224,224,3])  # Need to reshape your images.

    label = tf.cast(parsed_features['label'], tf.int32)  # Labels need to be int32

    label = tf.one_hot(label, depth=...)  # Convert our labels to one hot.   
return image, label 

We then pass this function to the dataset.map() method, which will execute it for us:

train_dataset = train_dataset.map(decode_tfrec, num_parallel_calls=4)