Feedforward Layer
This tutorial assumes that you understand the basic knowledge of Feedforward Neural Networks, including concepts of
 input neuron
 output neuron
 weight matrix
 bias
Initialize an Abstract Layer
A Feedforward Neural Network (FNN) consists of series of fullyconnected layers. These layers has some common properties so that we can write an abstract layer so that we donâ€™t have to write them again and again for different types of feedforward layer.
A FNN layer is defined by at least 4 parameters:

input dimension

output dimension

Weight matrix

Bias vector
With these 4 parameters, we can generate or load a layer as wish.
The following is an example implementation of __init__
method (here):
1class Layer(object): 2 """Abstract layer for Feedforward Neural Networks""" 3 4 def __init__(self, 5 in_dim, 6 out_dim, 7 layer_name="Layer", 8 W=None, 9 bias=None, 10 use_bias=True, 11 is_recursive=False, 12 **kwargs): 13 """Base Layer initalization 14 15 Parameters 16  17 in_dim : int 18 input dimension of the layer 19 out_dim : int 20 output dimension of the layer 21 W : matrix 22 weight matrix for the layer, the size should be (in_dim, out_dim), 23 if it is None, then the class will create one 24 bias : vector 25 bias vector for the layer, the size should be (out_dim), 26 if it is None, then the class will create one 27 """ 28 29 self.in_dim=in_dim; 30 self.out_dim=out_dim; 31 self.W=W; 32 self.bias=bias; 33 self.use_bias=use_bias; 34 self.is_recursive=is_recursive; 35 36 self.initialize(); 37 38 super(Layer, self).__init__(**kwargs);
Usually, we would like to initialize the weight when we create a layer. I used to initialize them in __init__
, however, this could be messy when you try to extend this class to a more complex one. Therefore, I wrote an initialize function to make initialization more flexible (here):
1 def initialize(self, weight_type="none"): 2 """Initialize weights and bias 3 4 Parameters 5  6 weight_type : string 7 type of weights: "none", "tanh", "sigmoid" 8 """ 9 10 if self.W==None: 11 self.W=util.init_weights("W", self.out_dim, self.in_dim, weight_type=weight_type); 12 13 if self.use_bias==True and self.bias==None: 14 self.bias=util.init_weights("bias", self.out_dim, weight_type=weight_type);
Since numpy
recognize a 1D array as row vector, therefore we prefer to use row based structure for our data. This means that each row is one piece of data. Hence, your data \(X\) is in size of number of number * size of sample
. And the linear transformation between input \(X\), output \(Y\), weights \(W\) and bias \(b\) is
A example of writing it is (here):
1 def apply_lin(self, X): 2 """Apply linear transformation 3 4 Parameters 5  6 X : matrix 7 input samples, the size is (number of cases, in_dim) 8 9 Returns 10  11 Y : matrix 12 output results, the size is (number of cases, out_dim); 13 """ 14 15 Y=T.dot(X, self.W); 16 17 if self.use_bias==True: 18 Y+=self.bias; 19 20 return Y;
In above code, I introduced a parameter use_bias
. If you set this to false
, then you will compute a set of linear equations that are not includes bias in Yaxis.
Surprisingly, there is not much work to write a general layer. The above 3 functions is actually enough to create a sufficient Feedforward Layer.