In order to build the encoder, we need to figure out how many pixels each MNIST image will have so that we can figure out the size of the input layer of the encoder. Each image from the MNIST dataset is 28 by 28 pixels, so we will reshape this matrix to a vector of 28 x 28 = 784 pixel values. We don't have to normalize the images of MNIST because they are already normalized.

Let's start off building our three components of the model. In this implementation, we will use a very simple architecture of a single hidden layer followed by ReLU activation, as shown in the following figure:

Let's go ahead and implement this simple encoder-decoder architecture according to the preceding explanation:

# The size of the encoding layer or the hidden layer.
encoding_layer_dim = 32 

img_size = mnist_dataset.train.images.shape[1]

# defining placeholder variables of the input and target values
inputs_values = tf.placeholder(tf.float32, (None, img_size), name="inputs_values")
targets_values = tf.placeholder(tf.float32, (None, img_size), name="targets_values")

# Defining an encoding layer which takes the input values and incode them.
encoding_layer = tf.layers.dense(inputs_values, encoding_layer_dim, activation=tf.nn.relu)

# Defining the logit layer, which is a fully-connected layer but without any activation applied to its output
logits_layer = tf.layers.dense(encoding_layer, img_size, activation=None)

# Adding a sigmoid layer after the logit layer
decoding_layer = tf.sigmoid(logits_layer, name = "decoding_layer")

# use the sigmoid cross entropy as a loss function
model_loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits_layer, labels=targets_values)

# Averaging the loss values accross the input data
model_cost = tf.reduce_mean(model_loss)

# Now we have a cost functiont that we need to optimize using Adam Optimizer
model_optimizier = tf.train.AdamOptimizer().minimize(model_cost)

Now we have defined our model and also used a binary cross-entropy since the images, pixels are already normalized.