Feedforward neural networks

Here x is an input, y is a “label”, v ∈ Rd is a parameter vector, and f(x, y) ∈ Rd is a feature vector that corresponds to a representation of the pair (x, y). Log-linear models have the advantage that the feature vector f(x, y) can include essentially any features of the pair (x, y). However, these features are generally designed by hand, and in practice this is a limitation. It can be laborious to define features by hand for complex problems such as language modeling, tagging, parsing, or machine translation. Neural networks essentially allow the representation itself to be learned. In practice, this can significantly decrease the amount of human engineering required in various applications. More importantly, empirical results across a broad set of domains have shown that learned representations in neural networks can give very significant improvements in accuracy over hand-engineered features. In the remainder of this note we first introduce multi-class feedforward networks. In a later note we will describe how these models can be trained, using stochastic gradient descent in conjunction with the backpropagation algorithm for calculation of gradients.