Introducing Neural Networks.

Figure 82. Neural networks first steps video

Neural networks 1st steps.mov

Neural networks are biologically inspired machine learning methods that can contain multiple algorithms. They are at the cutting-edge of machine learning, and offer extremely powerful approaches, so it’s worth getting a feel for what they do before tackling the other topics ahead of us.

The Perceptron.

Neural networks are modelled on how the brain works at its most basic level. Our thinking is a product of electro chemical activity. Thoughts, or more precisely ‘cognition’, occurs in ‘synapses’. A synapse is formed when two or more neurons are connected electrically. At the most basic level, a neuron makes a decision, and that decision depends on weighted influences.

Imagine you were considering whether to buy something or not. Two people told you ‘yes’, you should buy it, one person said ‘no, don’t buy it’. Chances are you’d go with the two “yes” answers.

A neuron works in a similar way. If it receives enough stimulus, a connect in is made, and that connection is called a synapse. We can mimic this with software, and doing so unleashes enormous machine learning capabilities. 

The basic building block of a neural network is the perceptron, which mimics how the synapse works.

Figure 88. The perceptron

As with a biological brain, machine synapses - perceptrons - can be chained, which means that neural networks can be extremely powerful.

Neural Networks Applied to a Problem.

Let’s look at how a neural network could be used to classifying food ingredients as unhealthy or otherwise. You will see that it is based on labels, features and vectors.

Figure 83. Binary classification problem for a neural network

Can you work out the rule, algorithm and how Ingredient ‘x’ should be categorised?

Ans – The rule is that any ingredient with fat or salt is given a vector of 0

Ingredient ’x’ has fat, so a vector of 0, therefore it is categorised as unhealthy. The algorithm is “If vector = 0, then ingredient = unhealthy”.

Let’s see how a neural network approach could handle a more challenging categorisation. Take the example of an Avocado. Avocados contain fat, as well as vitamins and fibre. So now the classification is a little harder.

Neural networks allow us to use multiple logical algorithms arranged in layers, which gives us more flexibility in the categorisation.

The layers below show the training data as an input layer, a set of algorithms that perform calculations in the middle (‘hidden’) layer, then an output layer which gives us a prediction or answer. This takes the form of a simple perceptron.

Figure 84. Structure of the basic neural network

Here we can say that to be classed as ’healthy, bad features (salt and fat) must be outnumbered by good features.

Using the logic “a or b & x + y = healthy” in the ‘hidden’ layer’s algorithm gives us a way to categorise something that is on the borderline.

Using neural Networks Layering to Make Predictions.

Figure 85. Neural network part 2 video

Neural network 2 720

The example above, Figure 84, we processed just one piece of data – an ingredient. But what happens if we want to process many pieces of data simultaneously?

The beauty of neural networks is that they can take many inputs at the same time.

In the example below, we will use a neural network approach to take many ingredients combined into a pizza, and then work out if the pizza overall is healthy or not.

We start with pizza ingredients with vectors indicating whether they are healthy (1) or not (0). These ingredients are chunked into the first set of ‘nodes’ starting with the size of the pizza, and finishing with drinks added to the meal.

The layers of the neural network contains ‘nodes’ – i.e. places where data is processed. In figure 86, nodes are either in circles or ovals.

It’s important at this point to understand that these nodes and layers exists in software. They are virtual – in other words, there isn’t a physical circuit as such, but defining the nodes and layers in this way provides a blueprint for the software.

For each input layer node there are 3 choices, e.g., the ‘Size’ node has the choice of small, medium and large. Each choice is given a vector number in the same way as we did in the previous example, ie small gets a ‘1’ whilst medium and large gets a 0.

In turn, the input nodes are chunked-up in groups of 3 into Factors.

The first layer of algorithms (Algorithm 1) adds up the numbers in the Input Layer vector column for each Factor, then passes that number to Algorithm 2.

Algorithm 2, adds up the numbers in each of the Algorithm 1 nodes and runs it through an algorithm which sets a threshold for classifying the pizza as “healthy” or otherwise.

From the output is the product of Algorithm 2.

Figure 86. Neural network with 'hidden layer' (Algorithms 1)

Now select your ingredients, run both sets of algorithms and classify your pizza.

Here’s an example answer.

Figure 87. Prediction

Error Correction.

To complete our introduction to machine learning, the final piece of the model is ’error correction’.

As we saw in the Healthy Pizza example that we explored earlier, we have a set of inputs, and we give each input a weight, which can be a positive or negative number. These weights are themselves weighed with algorithms in successive layers until we arrive at an outcome.

The reason neural networks are so powerful is because of the extent to which connections can be made. Millions of neurons arranged in layers, can make billions of connections, and each connection can use different ‘synaptic weighs’; which in turn opens up an almost infinite number of permutations.

The recent breakthrough in neural networks came with the relatively recent introduction of feedback. The advantage with feedback is that it can strengthen or weaken weightings in the neurons. In practice this means that categorisation gets increasingly accurate with the volume of data that the neural network is exposed to.

Figure 89. Neural network with error correction

The more layers in the neural network the 'deeper' the learning that happens.

When a machine learning instance is going through the learning process, there needs to be a way in which the system learns from errors.

In a neural network, error connection can be achieved a number of methods, and here we will look at a method called Back Propagation.

In a simple, yes/no, 1/0 scenario, we subtract the machine generated output from a known answer, and send the difference back into network to adjust algorithms. This is done repeatedly until the machine generated output is either 1 or 0, confirming or denying the known answer.

When training the model, we calculate the difference between the obtained values from the desired one, and send this difference back into the network to adjust the state of the ‘neurons”.

This process is repeated many times, and each iteration will produce increasingly precise outputs, calibrating each network's nodes, and refining its ability to process received data to make accurate predictions. Now we can see the 4 key parts of machine learning working together in the training process.

·      We take in input data

·      Pass this onto algorithms

·      Calculate the difference between the output and the known answer, and correct the error

·      Repeat until we get accurate predictions

Once the neural network correctly makes predictions, we can take it out of “learning mode” and feed it with the test data we want it to work on.

Applied Error Correction.

Now think of an example where you have directly experience error correction working to improve machine learning that you use daily.

One example of where you may regularly use machine learning with error correction, is Voice Recognition. In the greatly simplified model below we can visualise error correction in phone-based voice recognition.

Error correction will be used extensively in the learning process prior to the software being put on the phone, to train the software to correctly interpret different languages, accents and dialects.

When you get the phone, you’ll load the appropriate algorithms for your particular language and locale, but the software will still have some learning to do in order to fully adjust to your unique voice.

The more you talk into your voice recognition system, the more error correction it will go through until eventually it will work as smoothly as possible.

Figure 90. Simplified schematic for neural network applied to voice recognition