Neural Networks for beginners

Brain plasticity

2008-06-16T04:00:00.000-07:00

Neuroscience has altered considerably in the earlier 20 years. An example of change over phase is the theory of brain plasticity. Brain plasticity refers to the brain's ability to rewire itself, relocating information processing functions to different brain areas and/or neural networks. Two decades ago, it was assumed that brain networks were static after its early formation time. Now that belief has distorted. The analysis of brain plasticity has profound implications in human learning and behavior, and as such, for mental strength.

To better understand this idea, let's take a short tour of the human brain, neural networks, and the plastic potential therein.

Brains, Neurons and Networks

The brain is a multilayered parallel structure in which billions of neurons are interconnected and swap information through neural networks. In the brain, each neuron is connected to thousands of other neurons through synapses (specialised neuronal junctions). A connected neuron receives input from numerous other neurons, and when the input weight reaches a activation value, the neuron propagates an electrical signal that stimulates output through the ignition of a neurotransmitter (input to another neuron).

This electrochemical trade is the core of brain cell communication. It is also the premise for the formation of neural networks. These networks are produced during early childhood and are responsible for particular brain tasks, such as learning, pattern recognition and problem-solving. It was said that once neural networks were formed, they would remain 'hard-wired' or inflexible. However, inquiries in the past two decades has indicated that this is not the reality: our neural networks are in fact adaptive, flexible and responsive to change.

Rewiring is the Key

So what does it really mean to have a plastic brain? It has many implications to human behaviour and learning patterns. Primarily, it defies the old adage that "an old dog cannot learn new tricks". It is clear that with age, it becomes increasingly more complex to learn new things. However, the brain's ability to adapt to change perpetuates throughout an individual's lifetime.

A prominent case of neuroplasticity happened with a patient who spent 19 years in a coma. Terry Wallis, a 19 year old man from Massachusetts (US), woke up after spending 19 years in a minimally conscious state. When scientists scanned his brain combining PET (Positron Emission Tomography) and DTI (Diffusion Tensor Imaging) technologies, they found data that Wallis's brain had "developed new pathways and completely novel anatomical structures to re-establish functional connections, compensating for the brain pathways misplaced in the accident" (New Scientist, 03/07/2006).

Other cases, including stroke victims, people who have lost sensorial abilities (e.g. visually impaired) and individuals who have suffered cortical injuries show analogous conclusions after researchers have investigated how they have improved, or how the brain rewired itself to compensate for the injured areas and lost functions. The process of rewiring occurs when new connections (synapses) between neurons are formed and, if they prove to be favourable, they are likely to become more permanent and stabilised. This method allows the brain circuitry to be malleable to changes, or in other words, to form 'uncommon' networks under particular conditions.

Learning and Plasticity

Brain plasticity is not restricted to unplanned circumstances, such as accidents, brain traumas and other critical instances that require rewiring to re-establish functional connections. Learning is also a major beneficiary of brain plasticity. Studies with musicians and athletes have shown that particular areas of the brain responsible for 'fine' or 'specific' movements in certain parts of the body (e.g. the hands of a pianist or string musician) are in fact rewired for optimization. Once training becomes a routine, and particular actions are repeated over and over again, the tendency is that neuronal connections will become more permanent.

But there is more to it. Material contact is not a requirement when it comes to rewiring. Repeated thinking can also trigger a sequence of reactions which result in brain rewiring. Scientists have investigated the formation of synapses as a product of 'thinking about doing something' and found that, from a neuronal perspective, thinking can be as useful as doing. This evidence led to an interesting fusion of interests between Buddhist meditation (through the Dalai Lama's interest on the influence of the mind over the brain) and the scientific study on brain plasticity and the formation of neural networks. It seems that brain plasticity is a flexible topic as well as a stretchy concept.

Mind Your Thoughts

Learning and plasticity took center stage when collaborative research was conducted with lamas (Buddhist equivalent for priests or spiritual leaders). It seems that, as a result of ongoing meditation through a technique called Mindfulness (which aims to develop the person's control and awareness of thoughts and emotions), the lamas were 'more able' to attain emotional consider and to concentrate.

Some of these studies include experiments performed by Dr. Kabat-Zinn (who educated mindfulness to employees in a high-pressure biotech business and concluded that stress levels were optimized over a short period of time) and Dr. Ekman's tests involving emotional expression detections. "The mindfulness training focuses on learning to monitor the continuing sensations and thoughts more closely, both in sitting meditation and in activities like yoga exercises" (NY Times, 04/02/2003).

The benefits of meditation through brain rewiring, from a non-religious perspective, are becoming clearer and quite appealing. Currently, there are therapeutic techniques that mix mindfulness with other mainstream therapies such as Cognitive Behavior Therapy. These have proven particularly useful for cases of depression and anxiety, for example.

Stepping Into the Strange

Brain plasticity has become a major topic of study. As fresh scanning technologies allow scientists to monitor the formation of synapses under particular stimuli, and experiment with living organisms, the applications of this knowledge are getting a range of research fields. Some scientists have promoted the idea of using stimulation to increase learning, however, at a neurochemical level. Others like the idea of meditation and 'wishful thinking' to empower the procedure of learning and to optimize the performance of certain tasks.

This collaborative approach from representatives of a non-dogmatic religion such as Buddhism, cognitive researchers and neuroscientists seems to be opening an attractive scope on the notion of brain plasticity. How far will this go? Hard to say, but nevertheless: very interesting to mind.

Perceptron neural network

2008-05-22T22:31:00.000-07:00

Perceptrons are the simplest architecture to learn when studying Neural Networking. Picture you mind of a perceptron as a node of a wide, interconnected neural network, sort of like a data tree, although the neural network does not necessarily have to have a top and bottom sections. The connections among all the nodes not only show the relationship between the nodes but also transmit data and information, called a signal or impulse. The perceptron is a simple model of a neuron .

Since making connections of perceptrons into a neural structure is a bit complicated, let's take a perceptron by itself. A perceptron has a number of external input pattern, one internal input (called a bias), a threshhold, and one output. To the right, you can see a picture of a simple perceptron. It resembles a neuron.

Usually, the input values are boolean (just two possible values 1 and 0, true and false), but they can be any number. The output of the perceptron, however, is always boolean like a switch. When the output is on (has the value 1), the perceptron is said to be activated.

All of the inputs (including the bias) have weights attached to the input patterns that modify the input values to the neural network. The weight is just multiplied with the input.

The activation function is one of the key components of the perceptron as in the most common neural network architectures. It determines, based on the inputs, whether the perceptron activates or not. Basically, the perceptron takes all of the weighted input values and adds them together. If the sum is above or equal to some value (called the threshold) then the perceptron fires. Otherwise, the perceptron does not. So, it get active whenever the following equation is true (where w represents the weight, and there are n inputs):

The threshold is like a wall: if the "signal" has enough "energy" to jump over the wall, then it can keep going, but otherwise, it has to stop. Traditionally, the threshold value is represented either as the Greek letter theta (the symbol inside the circle in the picture above) or by a graphical symbol that looks like a square S:

The main feature of perceptrons is that they can be trained (or learn) to behave a certain way as all neural networks. One popular beginner's assignment is to have a perceptron model (that is, learn to be) a basic boolean function such as AND or OR. Perceptron learning is supervised, that is, you have to have something that the perceptron can imitate. So, the perceptron learns like this: it produces an output, compares the output to what the output should be, and then adjusts itself a little bit. After repeating this cycle enough times, the perceptron will have converged (a technical name for learned) to the correct behavior.

This learning method is called the delta rule, because of the way the perceptron checks its accuracy. The difference between the perceptron's output and the correct output is assigned the Greek letter delta, and the Weight i for Input i is altered like this (the i shows that the change is separate for each Weight, and each weight has its corresponding input):

Change in Weight i = Current Value of Input i × (Desired Output - Current Output)

This can be elegantly summed up to:

The delta rule works both if the perceptron's output is too large and if it is too small. The new Weight i is found simply by adding the change for Weight i to the current value of Weight i.

Interestingly, if you graph the possible inputs on different axes of a mathematical graph, with pluses for where the perceptron fires and minuses where the perceptron doesn't, the weights for the perceptron make up the equation of a line that separates the pluses and the minuses.

For instance, in the picture above, the pluses and minues represent the OR binary function. With a little bit of simple algebra, you can transform that equation in the diagram to the standard line form in which the weights can be seen clearly. (You get the following equation of the line if you take the firing equation and replace the "greater than or equal to" symbol with the equal sign).

This equation is significant, because single perceptron can only model functions whose graphical models are linearly separable. So, if there is no line (or plane, or hyperplane, etc. depending on the number of dimensions) that divides the fires and the non-fires (the pluses and minuses), then it isn't possible for the perceptron to learn to behave with that pattern of firing. For instance, the boolean function XOR is not linearly separable, so you can't model this boolean function with only one perceptron. The weight values just keep on shifting, and the perceptron never actually converges to one value.

So, by themselves, perceptrons are a bit limited, but that is their appeal. Perceptrons enable a pattern to be broken up into simpler parts that can each be modeled by a separate perceptron in a network. So, even though perceptrons are limited, they can be combined into one powerful network that can model a wide variety of patterns, such as XOR and many complex boolean expressions of more than one variable. These algorithms, however, are more complex in arrangement, and thus the learning function is slightly more complicated. For many problems (specifically, the linearly separable ones), a single perceptron will do, and the learning function for it is quite simple and easy to implement. The perceptron is an elegantly simple way to model a human neuron's behavior. All you need is the first two equations shown above.

A detailed explanations about perceptron neural network can be found at Perceptron and Adaline Neural Networks

Backpropagation Neural network

2008-05-07T21:29:00.001-07:00

Introduction

Before we begin, know that our goal is to give you as much useful information as we can fit on our page.

This condition focuses on a particular print of neural network exemplar, known as a "supply-forwards back-propagation network". This exemplar is painless to understand, and can be openly employed as a software simulation.

First we will argue the foremost concepts behind this print of NN, then we'll get into some of the more useful application dreams.

Complex problems

As we take the journey through the final part of this article, you can look back at the first part if you need any clarifications on what we have already learned.

The domain of neural netmachinery can be thought of as being allied to artificial intelligence, invent education, matching meansing, statistics, and other domains. The attraction of neural netmachinery is that they are best competent to solving the troubles that are the most strenuous to explain by traditional computational reasonings.

judge an aura meansing charge such as recognizing an everyday thing projected against a background of other things. This is a charge that even a small newborn's wits can explain in a few tenths of a following. But house a conventional series invent to execute as well is incredibly dense. However, that same newborn might NOT be clever of calculating 2+2=4, while the series invent explains it in a few nanofollowings.

A fundamental difference between the aura recognition riddle and the addition riddle is that the past is best explaind in a matching frame, while easy mathematics is best done seriesly. Neurobiologists think that the wits is alike to a massively matching analog mainframe, containing about 10^10 easy meansors which each expect a few millifollowings to counter to store. With neural network technology, we can use matching meansing reasonings to explain some authentic-world troubles where it is very strenuous to classify a conventional algorithm.

The Feed-Forward Neural Network form

If we think the creature wits to be the 'final' neural network, then beliefly we would like to develop a invent which copys the wits's performs. However, because of limits in our technology, we must mend for a greatly easyr invent. The evident line is to invent a small electronic invent which has a reassign perform alike to a biological neuron, and then link each neuron to many other neurons, with RLC netmachinery to copy the dendrites, axons, and synapses. This print of electronic exemplar is still instead dense to employ, and we may have strenuousy 'thinking' the network to do something effective. advance conslinets are wanted to make the invent more manageable. First, we change the linkivity between the neurons so that they are in patent layers, such that each neruon in one layer is linked to every neuron in the next layer. advance, we classify that suggests issue only in one focus across the network, and we simplify the neuron and synapse invent to perform as analog comparators being motivated by the other neurons through easy resistors. We now have a supply-forwards neural network exemplar that may actually be useful to develop and use.

Referring to numbers 1 and 2, the network performs as follows: Each neuron receives a suggest from the neurons in the preceding layer, and each of those suggests is multiplied by a part influence cost. The influenceed stores are summed, and agreed through a warning perform which scales the harvest to a preset stretch of costs. The harvest of the limiter is then advertise to all of the neurons in the next layer. So, to use the network to explain a riddle, we affect the store costs to the stores of the first layer, allocate the suggests to breed through the network, and read the harvest costs.

while the authentic uniqueness or 'intelligence' of the network exists in the costs of the influences between neurons, we essential a reasoning of adjusting the influences to explain a particular riddle. For this print of network, the most frequent education algorithm is called Back Propagation (BP). A BP network learns by example, that is, we must grant a education set that consists of some store examples and the known-rectify harvest for each container. So, we use these store-harvest examples to show the network what print of actions is likely, and the BP algorithm allocates the network to adapt.

The BP education means machinery in small iterative steps: one of the example containers is useful to the network, and the network produces some harvest based on the stream majesty of it's synaptic influences (primarily, the harvest will be haphazard). This harvest is compared to the known-good harvest, and a mean-squared mistake suggest is calculated. The mistake cost is then breedd backwards through the network, and small changes are made to the influences in each layer. The influence changes are calculated to lessen the mistake suggest for the container in problem. The full means is recurring for each of the example containers, then back to the first container again, and so on. The round is recurring awaiting the general mistake cost drops below some pre-determined threshold. At this item we say that the network has erudite the riddle "well enough" - the network will never rigidly learn the belief perform, but instead it will asymptotically line the belief perform.

When to use (or not!) a BP Neural Network result

A back-propagation neural network is only useful in certain situations. next are some guidelines on when you should use another line:

* Can you write down a issue chart or a formula that accurately describes the riddle? If so, then spike with a traditional programming reasoning.

* Is there a easy instance of hardware or software that already does what you want? If so, then the development time for a NN might not be value it.

* Do you want the performality to "evolve" in a focus that is not pre-classifyd? If so, then think with a Genetic Algorithm (that's another subject!).

* Do you have an painless way to cause a significant number of store/harvest examples of the beloved actions? If not, then you won't be able to line your NN to do something.

* Is the riddle is very "discrete"? Can the rectify answer can be found in a look-up graph of reasonable mass? A look-up graph is greatly easyr and more accurate.

* Are rigid numeric harvest costs expectd? NN's are not good at generous rigid numeric answers.

Conversely, here are some situations where a BP NN might be a good idea:

* A large total of store/harvest figures is existing, but you're not loyal how to concern it to the harvest.

* The riddle appears to have overwhelming denseity, but there is openly a blend.

* It is painless to invent a number of examples of the rectify actions.

* The blend to the riddle may change over time, inside the bounds of the given store and harvest parameters (i.e., nowadays 2+2=4, but in the outlook we may find that 2+2=3.8).

* Outputs can be "fuzzy", or non-numeric.

One of the most frequent applications of NNs is in aura meansing. Some examples would be: identifying hand-printed characters; matching a photograph of a part's face with a different photo in a figuresbase; executeing figures compression on an aura with token harm of content. Other applications could be: declare recognition; RADAR signature testing; cattle souk prediction. All of these troubles implicate large totals of figures, and dense relationships between the different parameters.

It is important to recall that with a NN blend, you do not have to understand the blend at all! This is a foremost gain of NN linees. With more traditional techniques, you must understand the stores, and the algorithms, and the harvests in great describe, to have any prospect of employing something that machinery. With a NN, you merely show it: "this is the rectify harvest, given this store". With an adequate total of lineing, the network will mimic the perform that you are demonstrating. advance, with a NN, it is OK to affect some stores that shot out to be irrelevant to the blend - during the lineing means, the network will learn to snub any stores that don't contribute to the harvest. Conversely, if you authority out some vital stores, then you will find out because the network will bomb to meet on a blend.