This tutorial assumes that you understand the basic knowledge of Feedforward Neural Networks, including concepts of

  • input neuron
  • output neuron
  • weight matrix
  • bias

Initialize an Abstract Layer

A Feedforward Neural Network (FNN) consists of series of fully-connected layers. These layers has some common properties so that we can write an abstract layer so that we don’t have to write them again and again for different types of feedforward layer.

A FNN layer is defined by at least 4 parameters:

  • input dimension

  • output dimension

  • Weight matrix

  • Bias vector

With these 4 parameters, we can generate or load a layer as wish.

The following is an example implementation of __init__ method (here):

 1class Layer(object):
 2    """Abstract layer for Feed-forward Neural Networks"""
 4    def __init__(self,
 5                 in_dim,
 6                 out_dim,
 7                 layer_name="Layer",
 8                 W=None,
 9                 bias=None,
10                 use_bias=True,
11                 is_recursive=False,
12                 **kwargs):
13        """Base Layer initalization
15        Parameters
16        ----------
17        in_dim : int
18            input dimension of the layer
19        out_dim : int
20            output dimension of the layer
21        W : matrix
22            weight matrix for the layer, the size should be (in_dim, out_dim),
23            if it is None, then the class will create one
24        bias : vector
25            bias vector for the layer, the size should be (out_dim),
26            if it is None, then the class will create one
27        """
29        self.in_dim=in_dim;
30        self.out_dim=out_dim;
31        self.W=W;
32        self.bias=bias;
33        self.use_bias=use_bias;
34        self.is_recursive=is_recursive;
36        self.initialize();
38        super(Layer, self).__init__(**kwargs);

Usually, we would like to initialize the weight when we create a layer. I used to initialize them in __init__, however, this could be messy when you try to extend this class to a more complex one. Therefore, I wrote an initialize function to make initialization more flexible (here):

 1    def initialize(self, weight_type="none"):
 2        """Initialize weights and bias
 4        Parameters
 5        ----------
 6        weight_type : string
 7            type of weights: "none", "tanh", "sigmoid"
 8        """
10        if self.W==None:
11            self.W=util.init_weights("W", self.out_dim, self.in_dim, weight_type=weight_type);
13        if self.use_bias==True and self.bias==None:
14            self.bias=util.init_weights("bias", self.out_dim, weight_type=weight_type);

Since numpy recognize a 1-D array as row vector, therefore we prefer to use row based structure for our data. This means that each row is one piece of data. Hence, your data \(X\) is in size of number of number * size of sample. And the linear transformation between input \(X\), output \(Y\), weights \(W\) and bias \(b\) is

A example of writing it is (here):

 1    def apply_lin(self, X):
 2        """Apply linear transformation
 4        Parameters
 5        ----------
 6        X : matrix
 7            input samples, the size is (number of cases, in_dim)
 9        Returns
10        -------
11        Y : matrix
12            output results, the size is (number of cases, out_dim);
13        """
15, self.W);
17        if self.use_bias==True:
18            Y+=self.bias;
20        return Y;

In above code, I introduced a parameter use_bias. If you set this to false, then you will compute a set of linear equations that are not includes bias in Y-axis.

Surprisingly, there is not much work to write a general layer. The above 3 functions is actually enough to create a sufficient Feedforward Layer.