2023-02-17

RNN

What is RNN

RNN stands for Recurrent Neural Network. RNN is a type of neural network for processing sequences of input data, such as time series data or natural language, which is It is suitable for processing continuous input data, such as time-series data or natural language.

RNNs can take a sequence of input, such as time-series data, and process it with temporal dependencies in mind; RNNs, unlike traditional neural networks, can retain state for their input and convey the previous state to the next step. This allows the RNN to retain past information.

In the following RNN diagram, part of the output of the neural net at time $t$ is used as input to the neural net at time $t+1$ . In this way, it allows for considerations regarding time series.

RNN | 2
Understanding LSTM Networks

The main architectures of RNNs include simple RNNs, LSTMs, and GRUs. These architectures can store historical input data and combine it with the current input to produce output. These architectures are used for tasks such as time series data prediction, natural language processing, speech recognition, and image captioning.

RNN | 3
A simple overview of RNN, LSTM and Attention Mechanism

Hidden state

The hidden state is a term used in fields such as machine learning and natural language processing to refer to unobserved states that a model maintains internally.

In RNNs, hidden states are computed from past inputs and their own outputs and are used to predict and process information about the next input. It converts the input sequence into a single vector and updates the hidden state based on it. The presence of hidden states makes the model more expressive for more complex problems and allows predictions that take into account the concept of time series.

The hidden state at $t$ is represented by a function of the hidden state at $t-1$ and the input $x_t$ .

h_t = g(h_{t-1}, x_t)

Since the output of a recurrent neuron at a given time step $t$ is a function of its input up to time step $t-1$ , this mechanism can be considered a kind of memory. The part of the neural network that has the concept of storing state, even partially, across time steps is commonly referred to as a memory cell.

Hidden State
Understanding LSTM Networks

Difference between hidden layer and hidden state

Hidden state and hidden layer are terms used in deep learning that are similar concepts, but have slightly different meanings.

Hidden state is a term used in models such as RNNs and LSTMs to describe the internal state, including the output at the previous time. in RNNs, hidden state is used to process time series data because it affects the input at the next time. In LSTMs, similarly, hidden state is used to preserve past information. The hidden layer is used to process time-series data, as it affects the input of the next time.

Hidden layers, on the other hand, are a term used in neural networks such as multilayer perceptrons (MLPs) to refer to the intermediate layer between the input and output layers. The hidden layer consists of multiple neurons, each of which receives signals from the previous layer, computes a weighted sum, and produces an output via an activation function. When there are multiple hidden layers, the output of each hidden layer is input to the next hidden layer or output layer.

In other words, hidden states and hidden layers are both concepts that represent internal states in machine learning, but they are used in different contexts and for different purposes. Hidden states are used to process time series data, while hidden layers represent the middle layer of a typical neural network.

RNN challenges

A major issue that makes RNN training very slow and inefficient is the gradient vanishing problem The process for a Feed Forward neural network is as follows

output some result in the forward pass
use that result to compute a loss value
use that loss value to back propagate and compute gradients with respect to the weights
back propagate these gradients with respect to the weights to fine tune the weights and improve the performance of the network

Because the weights are manipulated according to the previous layer, the small gradients tend to decrease significantly with each layer change and become very close to zero, resulting in poor learning in the early layers and overall ineffective learning.

Hidden State | 3
A simple overview of RNN, LSTM and Attention Mechanism

Thus, if the gradient disappears, the RNN will not be able to learn long-range dependencies across time steps well. In other words, even if the initial input of a sequence is important to the overall context, it will not be important. Therefore, it cannot learn long sequences, resulting in short-term memory.

Hidden State | 4
Understanding LSTM Networks

RNN Implementation

Here are some examples of RNN implementations using PyTorch and Keras.

PyTorch

The following is a basic example of using PyTorch to describe an RNN model. This example uses a simple RNN architecture to perform the task of classifying words.

import torch
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size
        self.rnn = nn.RNN(input_size, hidden_size)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, input):
        hidden = torch.zeros(1, 1, self.hidden_size)
        output, hidden = self.rnn(input, hidden)
        output = self.fc(output[-1, :, :])
        return output

This model takes in words of size input_size, passes them through an RNN with a hidden layer size of hidden_size, and finally generates a classification output of size output_size. The nn.RNN class is the built-in RNN layer in PyTorch, and nn.Linear represents a fully connected layer.

The forward method performs the forward pass on the given input. First, the hidden layer is initialized. Then, the input and current hidden state are passed to the RNN to obtain the output and new hidden state. Finally, only the output from the last step is taken and passed through the fully connected layer to generate the classification output.

RNN

What is RNN

Hidden state

Difference between hidden layer and hidden state

RNN challenges

RNN Implementation

PyTorch

Keras

References

Word Embeddings

Attention

Ryusei Kakujo