Neural networks for beginners

Hebbian Learning – The Value of Repetition

drios — Sun, 12 Sep 2010 17:59:00 +0000

You have always heard that “practice makes perfect.” Have you wondered why? It might just be related to the synaptic plasticity of the brain. How many times does a thought need to be repeated before it becomes sufficient hard wired into the brain?

In 1949, Canadian psychologist Donald Hebb postulated a theory in which he said that “the persistence or repetition of a reverberatory activity tends to induce lasting cellular changes that add to its stability.” Another way of saying this is “cells that fire together, wire together.” Hebbian Learning is a theory that explains that some types of associative learning in which simultaneous activation of cells leads to increases in synaptic strength. Indeed, this may explain why repeated thought or practice strengthens the hardwiring of neurons in the association areas of the various lobes of the brain.

Biologically this means that a dominant thought created through our will power will stimulate new synaptic connections in our brain. Once these connections are made, repetition of the same thought (or action) will stimulate more corresponding connections. These redundant connections become engrams, which are holographic stores of memories. These engrams involve a network of connections which facilitate synchronized synaptic firing thus producing a more efficient expression of a given thought. The more a thought is held, the easier for that thought remembered or activated. For instance, think of acquiring a new physical skill such as dancing or the martial arts or learning a new language.

Conversely there are thoughts or memories that have reached that level of engram efficiency, but are no longer rationally desired. This could be a phobic memory that somehow is hardwired into our survival mechanism. This means that this gestalt (a collection of memories) is connected to the hypothalamus and pituitary and thus creates neuropeptides, which in turn encode this memory at the cellular level. Therefore, any contrary thought will be resisted as our body will have a defensive reaction to the contrary feeling. So, how to be rid our self of these unwanted thoughts? How can be get out of our way?

Again, our will power is a key factor. But, often this is not enough. Obviously, if you follow the postulate of Hebbian Learning, you would say “use it or lose it.” If you could select a contrary (hopefully positive) thought, make it as vivid as possible, and have it recurring, you will biologically rewire your brain to develop new neural pathways while weakening the unwanted neural networks thought disuse.

Hypnosis is a great tool for helping bypass resistance and to install new thought patterns. However, repetition is still the key. Repeated hypnotherapy or self-hypnosis sessions can be used effectively to create new neural networks and to allow unwanted networks to atrophy.

Tim Brunson, PhD

The International Hypnosis Research Institute is a member supported project involving integrative health care specialists from around the world. We provide information and educational resources to clinicians. Dr. Brunson is the author of over 150 self-help and clinical CD’s and MP3′s.

Reference

Is Artificial Intellgience Possible?

drios — Sun, 12 Sep 2010 17:54:00 +0000

By Tommy Connolly

“Artificial Intelligence has been brain-dead since the 1970s.” This rather ostentatious remark made by Marvin Minsky co-founder of the world-famous MIT Artificial Intelligence Laboratory, was referring to the fact that researchers have been primarily concerned on small facets of machine intelligence as opposed to looking at the problem as a whole. This article examines the contemporary issues of artificial intelligence (AI) looking at the current status of the AI field together with potent arguments provided by leading experts to illustrate whether AI is an impossible concept to obtain.

Because of the scope and ambition, artificial intelligence defies simple definition. Initially AI was defined as “the science of making machines do things that would require intelligence if done by men”. This somewhat meaningless definition shows how AI is still a young discipline and similar early definitions have been shaped by technological and theoretical progress made in the subject. So for the time being, a good general definition that illustrates the future challenges in the AI field was made by the American Association for Artificial Intelligence (AAAI) clarifying that AI is the “scientific understanding of the mechanisms underlying thought and intelligent behaviour and their embodiment in machines”.

The term “artificial intelligence” was first coined by John McCarthy at a Conference at Dartmouth College, New Hampshire, in 1956, but the concept of machine intelligence is in fact much older. In ancient Greek mythology the smith-god, Hephaestus, is credited with making Talos, a “bull-headed” bronze man who guarded Crete for King Minos by patrolling the island terrifying off impostors. Similarly in the 13th century mechanical talking heads were said to have been created to scare intruders, with Albert the Great and Roger Bacon reputedly among the owners. However, it is only in the last 50 years that AI has really begun to pervade popular culture. Our fascination with “thinking machines” is obvious, but has been wrongfully distorted by the science-fiction connotations seen in literature, film and television.

In reality the AI field is far from creating the sentient beings seen in the media, yet this does not imply that successful progress has not been made. AI has been a rich branch of research for 50 years and many famed theorists have contributed to the field, but one computer pioneer that has shared his thoughts at the beginning and still remains timely in both his assessment and arguments is British mathematician Alan Turing. In the 1950s Turing published a paper called Computing Machinery and Intelligence in which he proposed an empirical test that identifies an intelligent behaviour “when there is no discernible difference between the conversation generated by the machine and that of an intelligent person.” The Turing test measures the performance of an allegedly intelligent machine against that of a human being and is arguably one of the best evaluation experiments at this present time. The Turing test, also referred to as the “imitation game” is carried out by having a knowledgeable human interrogator engage in a natural language conversation with two other participants, one a human the other the “intelligent” machine communicating entirely with textual messages. If the judge cannot reliably identify which is which, it is said that the machine has passed and is therefore intelligent. Although the test has a number of justifiable criticisms such as not being able to test perceptual skills or manual dexterity it is a great accomplishment that the machine can converse like a human and can cause a human to subjectively evaluate it as humanly intelligent by conversation alone.

Many theorist have disputed the Turing Test as an acceptable means of proving artificial intelligence, an argument posed by Professor Jefferson Lister states, “not until a machine can write a sonnet or compose a concerto because of thoughts and emotions felt, and not by the chance fall of symbols, could we agree that machine equals brain”. Turing replied by saying “that we have no way of knowing that any individual other than ourselves experiences emotions and that therefore we should accept the test.” However Lister did have a valid point to make, developing an artificial consciousness. Intelligent machines already exist that are autonomous; they can learn, communicate and teach each other, but creating an artificial intuition, a consciousness, “is the holy grail of artificial intelligence.” When modelling AI on the human mind many illogical paradoxes surface and you begin to see how the complexity of the brain has been underestimated and why simulating it has not be as straightforward as experts believed in the 1950’s. The problem with human beings is that they are not algorithmic creatures; they prefer to use heuristic shortcuts and analogies to situations well known. However, this is a psychological implication, “it is not that people are smarter then explicit algorithms, but that they are sloppy and yet do well in most cases.”

The phenomenon of consciousness has caught the attention of many Philosophers and Scientists throughout history and innumerable papers and books have been published devoted to the subject. However, no other biological singularity has remained so resistant to scientific evidence and “persistently ensnarled in fundamental philosophical and semantic tangles.” Under ordinary circumstances, we have little difficulty in determining when other people lose or regain consciousness and as long as we avoid describing it, the phenomenon remains intuitively clear. Most Computer Scientists believe that the consciousness was an evolutionary “add-on” and can therefore be algorithmically modelled. Yet many recent claims oppose this theory. Sir Roger Penrose, an English mathematical physicist, argues that the rational processes of the human mind are not completely algorithmic and thus transcends computation and Professor Stuart Hameroff’s proposal that consciousness emerges as a macroscopic quantum state from a critical level of coherence of quantum level events in and around cytoskeletal microtubules within neurons. Although these are all theories with not much or no empirical evidence, it is still important to consider each of them because it is vital that we understand the human mind before we can duplicate it.

Another key problem with duplicating the human mind is how to incorporate the various transitional states of consciousness such as REM sleep, hypnosis, drug influence and some psychopathological states within a new paradigm. If these states are removed from the design due to their complexity or irrelevancy in a computer then it should be pointed out that perhaps consciousness cannot be artificially imitated because these altered states have a biophysical significance for the functionality of the mind.

If consciousness is not algorithmic, then how is it created? Obviously we do not know. Scientists who are interested in subjective awareness study the objective facts of neurology and behaviour and have shed new light on how our nervous system processes and discriminates among stimuli. But although such sensory mechanisms are necessary for consciousness, it does not help to unlock the secrets of the cognitive mind as we can perceive things and respond to them without being aware of them. A prime example of this is sleepwalking. When sleepwalking occurs (Sleepwalking comprises approximately 25 percent of all children and 7 percent of adults) many of the victims carry out dangerous or stupid tasks, yet some individuals carry out complicated, distinctively human-like tasks, such as driving a car. One may dispute whether sleepwalkers are really unconscious or not, but if it is in fact true that the individuals have no awareness or recollection of what happened during their sleepwalking episode, then perhaps here is the key to the cognitive mind. Sleepwalking suggests at least two general behavioural deficiencies associated with the absence of consciousness in humans. The first is a deficiency in social skills. Sleepwalkers typically ignore the people they encounter, and the “rare interactions that occur are perfunctory and clumsy, or even violent.” The other major deficit in sleepwalking behaviour is linguistics. Most sleepwalkers respond to verbal stimuli with only grunts or monosyllables, or make no response at all. These two apparent deficiencies may be significant. Sleepwalkers luse of protolanguage; short, grammar-free utterances with referential meaning but lack syntax, may illustrate that the consciousness is a social adaptation and that other animals do not lack understanding or sensation, but that they lack language skills and therefore cannot reflect on their sensations and become self-aware. In principle Francis Crick, co-discover of double helix DNA structure, believed this hypotheses. After he and James Watson solved the mechanism of inheritance, Crick moved to neuroscience and spent the rest of his trying to answer the biggest biological question; what is the consciousness? Working closely with Christof Koch, he published his final paper in the Philosophical Transactions of the Royal Society of London and in it he proposed that an obscure part of the brain, the claustrum, acts like a conductor of an orchestra and “binds” vision, olfaction, somatic sensation, together with the amygdala and other neuronal processing for the unification of thought and emotion. And the fact that all mammals have a claustrum means that it is possible that other animals have high intelligence.

So how different are the minds of animals in comparison to our own? Can their minds be algorithmically simulated? Many Scientists are reluctant to discuss animal intelligence as it is not an observable property and nothing can be perceived without reason and therefore there is not much published research on the matter. But, by avoiding the comparison of some human mental states to other animals, we are impeding the use of a comparative method that may unravel the secrets of the cognitive mind. However primates and cetacean have been considered by some to be extremely intelligent creatures, second only to humans. Their exalted status in the animal kingdom has lead to their involvement in almost all of published experiments related to animal intelligence. These experiments coupled with analysis of primate and cetacean’s brain structure has lead to many theories as to the development of higher intelligence as a trait. Although these theories seem to be plausible, there is some controversy over the degree to which non-human studies can be used to infer about the structure of human intelligence.

By many of the physical methods of comparing intelligence, such as measuring the brain size to body size ratio, cetacean surpass non-human primates and even rival human beings. For example “dolphins have a cerebral cortex which is about 40% larger a human being. Their cortex is also stratified in much the same way as humans. The frontal lobe of dolphins is also developed to a level comparable to humans. In addition the parietal lobe of dolphins which “makes sense of the senses” is larger than the human parietal and frontal lobes combined. The similarities do not end there; most cetaceans have large and well-developed temporal lobes which contain sections equivalent to Broca’s and Wernicke’s areas in humans.”

Dolphins exhibit complex behaviours; they have a social hierarchy, they demonstrate the ability to learn complex tricks, when scavenging for food on the sea floor, some dolphins have been seen tearing off pieces of sponge and wrapping them around their “bottle nose” to prevent abrasions; illustrating yet another complex cognitive process thought to be limited to the great apes, they apparently communicate by emitting two very distinct kinds of acoustic signals, which we call whistles and clicks and lastly dolphins do not use sex purely for procreative purposes. Some dolphins have been recorded having homosexual sex, which demonstrates that they must have some consciousness. Dolphins have a different brain structure then humans that could perhaps be algorithmic simulated. One example of their dissimilar brain structure and intelligence is their sleep technique. While most mammals and birds show signs of rapid REM (Rapid Eye Movement) sleep, reptiles and cold-blooded animals do not. REM sleep stimulates the brain regions used in learning and is often associated with dreaming. The fact that cold-blooded animals do not have REM sleep could be enough evidence to suggest that they are not conscious and therefore their brains can definitely be emulated. Furthermore, warm-blood creatures display signs of REM sleep, and thus dream and therefore must have some environmental awareness. However, dolphins sleep unihemispherically, they are “conscious” breathers, and if fall asleep they could drown. Evolution has solved this problem by letting one half of its brain sleep at a time. As dolphins utilise this technique, they lack REM sleep and therefore a high intelligence, perhaps consciousness, is possible that does not incorporate the transitional states mentioned earlier.

The evidence for animal consciousness is indirect. But so is the evidence for the big bang, neutrinos, or human evolution. As in any event, such unusual assertions must be subject to rigorous scientific procedure, before they can be accepted as even vague possibilities. Intriguing, but more proof is required. However merely because we do not understand something does not mean that it is false – or not. Studying other animal minds is a useful comparative method and could even lead to the creation of artificial intelligence (that does not include irrelevant transitional states for an artificial entity), based on a model not as complex as our own. Still the central point being illustrated is how ignorant our understanding of the human brain, or any other brain is and how one day a concrete theory can change thanks to enlightening findings.

Furthermore, an analogous incident that exemplifies this argument happened in 1847, when an Irish workman, Phineas Cage, shed new light on the field of neuroscience when a rock blasting accident sent an iron rod through the frontal region of his brain. Miraculously enough, he survived the incident, but even more astonishing to the science community at the time were the marked changes in Cage’s personality after the rode punctured his brain. Where before Cage was characterized by his mild mannered nature, he had now become aggressive, rude and “indulging in the grossest profanity, which was not previously his custom, manifesting but little deference for his fellows, impatient of restraint or advice when it conflicts with his desires” according to the Boston physician Harlow in 1868. However, Cage sustained no impairment with regards to his intelligence or memory.

The serendipity of the Phineas Cage incident demonstrates how architecturally robust the structure of the brain is and by comparison how rigid a computer is. All mechanical systems and algorithms would stop functioning correctly or completely if an iron rod punctured them, that is with the exception of artificial neural systems and their distributed parallel structure. In the last decade AI has began to resurge thanks to the promising approach of artificial neural systems.

Artificial neural systems or simply neural networks are modelled on the logical associations made by the human brain, they are based on mathematical models that accumulate data, or “knowledge,” based on parameters set by administrators. Once the network is “trained” to recognize these parameters, it can make an evaluation, reach a conclusion and take action. In the 1980s, neural networks became widely used with the backpropagation algorithm, first described by Paul John Werbos in 1974. The 1990s marked major achievements in many areas of AI and demonstrations of various applications. Most notably in 1997, IBM’s Deep Blue supercomputer defeated the world chess champion Garry Kasparov. After the match Kasparov was quoted as saying the computer played “like a god.”

That chess match and all its implications raised profound questions about neural networks. Many saw it as evidence that true artificial intelligence had finally been achieved. After all, “a man was beaten by a computer in a game of wits.” But it is one thing to program a computer to solve the kind of complex mathematical problems found in chess. It is quite another for a computer to make logical deductions and decisions on its own.

Using neural networks, to emulate brain function, provides many positive properties including parallel functioning, relatively quick realisation of complicated tasks, distributed information, weak computation changes due to network damage (Phineas Cage), as well as learning abilities, i.e. adaptation upon changes in environment and improvement based on experience. These beneficial properties of neural networks have inspired many scientists to propose them as a solution for most problems, so with a sufficiently large network and adequate training, the networks could accomplish many arbitrary tasks, without knowing a detailed mathematical algorithm of the problem. Currently, the remarkable ability of neural networks is best demonstrated by the ability of Honda’s Asimo humanoid robot that cannot just walk and dance, but even ride a bicycle. Asimo, an acronym for Advanced Step in Innovative Mobility, has 16 flexible joints, requiring a four-processor computer to control its movement and balance. Its exceptional human-like mobility, are only possible because the neural networks that are connected to the robot’s motion and positional sensors and control its ‘muscle’ actuators are capable of being ‘taught’ to do a particular activity.

The significance of this sort of robot motion control is the virtual impossibility of a programmer being able to actually create a set of detailed instructions for walking or riding a bicycle, instructions which could then be built into a control program. The learning ability of the neural network overcomes the need to precisely define these instructions. However, despite the impressive performance of the neural networks, Asimo still cannot think for itself and its behaviour is still firmly anchored on the lower-end of the intelligent spectrum, such as reaction and regulation.

Neural networks are slowly finding there way into the commercial world. Recently, Siemens launched a new fire detector that uses a number of different sensors and a neural network to determine whether the combination of sensor readings are from a fire or just part of the normal room environment such as dust. Over fifty percent of fire call-outs are false and of these well over half are due to fire detectors being triggered by everyday activities as opposed to actual fires, so this is clearly a beneficial use of the paradigm.

But are there limitations to the capabilities of neural networks or will they be the solution to creating strong-AI? Artificial neural networks are biologically inspired but that does not mean that they are necessarily biologically plausible. Many Scientists have published their thoughts on the intrinsic limitations of using neural networks; one book that received high exposure within the Computer Scientist community in 1969 was Perceptron by Minsky and Papert. Perceptron brought clarity to the limitations of neural networks, although many scientists were aware of limited ability of an incomplex perceptron to classify patterns, Minsky’s and Papert’s approach of finding “what are neural networks good for?” illustrated what is impeding future development of neural networks. Within its time period Perceptron was exceptionally constructive and its identifiable content gave the impetus for later research that conquered some of the depicted computational problems restricting the model. An example is the exclusive-or problem. The exclusive-or problem contains four patterns of two inputs each; a pattern is a positive member of a set if either one of the input bits is on, but not both. Thus, changing the input pattern by one-bit changes the classification of the pattern. This is the simplest example of a linearly inseparable problem. A perceptron using linear threshold functions requires a layer of internal units to solve this problem, and since the connections between the input and internal units could not be trained, a perceptron could not learn this classification. Eventually this restriction was solved by incorporating extra “hidden” layers. Although advances in neural network research have solved many of the limitations identified by Minsky and Papert, numerous still remain such as networks using linear threshold units still violate the limited order constraint when faced with linearly inseparable problems Additionally, the scaling of weights as the size of the problem space increases remains an issue.

It is clear that the dismissive views about neural networks disseminated by Minsky, Papert and many other Computer Scientists have some evidential support, but still many researchers have ignored their claims and refused to abandon this biologically inspired system.

There have been several recent advances in artificial neural networks by integrating other specialised theories into the multi-layered structure in an attempt to improve the system methodology and move one step closer to creating strong-AI. One promising area is the integration of fuzzy logic. invented by Professor Lotfi Zadeh. Other admirable algorithmic ideas include quantum inspired neural networks (QUINNs) and “network cavitations” proposed by S.L.Thaler.

The history of artificial intelligence is replete with theories and failed attempts. It is in inevitable that the discipline will progress with technological and scientific discoveries, but will they ever reach the final hurdle?

Tommy Connolly, Undergraduate at the University of Exeter reading Computer Science
Reference

Iris Recognition Using Neural Networks

drios — Sun, 12 Sep 2010 17:50:00 +0000

By Parviz Eshaghi

Introduction

There are variable ways of human verification through out the world, as it is of great importance for all organizations, and different centers. Nowadays, the most important ways of human verification are recognition via DNA, face, fingerprint, signature, speech, and iris.

Among all, one of the recent, reliable, and technological methods is iris recognition which is practiced by some organizations today, and its wide usage in the future is of no doubt. Iris is a non identical organism made of colorful muscles including robots with shaped lines. These lines are the main causes of making every one’s iris non identical. Even the irises of a pair of eyes of one person are completely different from one another. Even in the case of identical twins irises are completely different. Each iris is specialized by very narrow lines, rakes, and vessels in different people. The precision of identification via iris is increased by using more and more details. It has been proven that iris patterns are never changed nearly from the time the child is one year old through out all his life.

Over the past few years there has been considerable interest in the development of neural network based pattern recognition systems, because of their ability to classify data. The kind of neural network practiced by the researcher is the Learning Vector Quantization which is a competitive network functional in the field of classification of the patterns. The iris images prepared as a database, is in the form of PNG (portable network graphics) pattern, meanwhile they must be preprocessed through which the boundary of the iris is recognized and their features are extracted. For doing so, edge detection is done by the usage of Canny approach. For more diverse and feature extraction of iris images DCT transform is practiced.

2. Feature Extraction

For increasing the precision of our verification of iris system we should extract the features so that they contain the main items of the images for comparison and identification. The extracted features should be in a way that cause the least of flaw in the output of the system and in the ideal condition the output flaw of the system should be zero. The useful features which should be extracted are obtained through edge detection in the first step and the in next step we use DCT transform.

2.1 Edge Detection

The first step locates the iris outer boundary, i.e. border between the iris and the sclera. This is done by performing edge detection on the gray scale iris image. In this work, the edges of the irises are detected using the “Canny method” which finds edges by finding local maxima of the gradient. The gradient is calculated using the derivative of a Gaussian filter. The method uses two thresholds, to detect strong and weak edges, and includes the weak edges in the output only if they are connected to strong edges. This method is robust to additive noise, and able to detect “true” weak edges.

Although certain literature has considered the detection of ideal step edges, the edges obtained from natural images are usually not at all ideal step edges. Instead they are normally affected by one or several of these effects: focal blur caused by a finite depth-of-field and finite point spread function, penumbral blur caused by shadows created by light sources of non-zero radius, shading at a smooth object edge, and local specularities or inter reflections in the vicinity of object edges.

2.1.1 Canny method

The Canny edge detection algorithm is known to many as the optimal edge detector. Canny’s intentions were to enhance the many edge detectors already out at the time he started his work. He was very successful in achieving his goal and his ideas and methods can be found in his paper, “A Computational Approach to Edge Detection”. In his paper, he followed a list of criteria to improve current methods of edge detection. The first and most obvious is low error rate. It is important that edges existing in images should not be missed and that there be NO responses to non-edges. The second criterion is that the edge points be well localized. In other words, the distance between the edge pixels as found by the detector and the actual edge is to be at a minimum. A third criterion is to have only one response to a single edge. This was implemented because the first 2 were not substantial enough to completely eliminate the possibility of multiple responses to an edge.

The Canny operator works in a multi-stage process. First of all the image is smoothed by Gaussian convolution. Then a simple 2-D first derivative operator (somewhat like the Roberts Cross) is applied to the smoothed image to highlight regions of the image with important spatial derivatives. Edges give rise to ridges in the gradient magnitude image. The algorithm then tracks along the top of these ridges and sets to zero all pixels that are not actually on the ridge top so as to give a thin line in the output, a process known as non-maximal suppression. The tracking process exhibits hysteresis controlled by two thresholds: T1 and T2, with T1 > T2. Tracking can only begin at a point on a ridge higher than T1. Tracking then continues in both directions out from that point until the height of the ridge falls below T2. This hysteresis helps to ensure that noisy edges are not broken up into multiple edge fragments.

2.2 Discrete Cosine Transform

Like any Fourier-related transform, discrete cosine transforms (DCTs) express a function or a signal in terms of a sum of sinusoids with different frequencies and amplitudes. Like the discrete Fourier transform (DFT), a DCT operates on a function at a finite number of discrete data points. The obvious distinction between a DCT and a DFT is that the former uses only cosine functions, while the latter uses both cosines and sinusoids (in the form of complex exponentials). However, this visible difference is merely a consequence of a deeper distinction: a DCT implies different boundary conditions than the DFT or other related transforms.

The Fourier-related transforms that operate on a function over a finite domain, such as the DFT or DCT or a Fourier series, can be thought of as implicitly defining an extension of that function outside the domain. That is, once you write a function f(x) as a sum of sinusoids, you can evaluate that sum at any x, even for x where the original f(x) was not specified. The DFT, like the Fourier series, implies a periodic extension of the original function. A DCT, like a cosine transform, implies an even extension of the original function.

A discrete cosine transform (DCT) expresses a sequence of finitely many data points in terms of a sum of cosine functions oscillating at different frequencies. DCTs are important to numerous applications in science and engineering, from lossy compression of audio and images (where small high-frequency components can be discarded), to spectral methods for the numerical solution of partial differential equations. The use of cosine rather than sine functions is critical in these applications: for compression, it turns out that cosine functions are much more efficient (as explained below, fewer are needed to approximate a typical signal), whereas for differential equations the cosines express a particular choice of boundary conditions.

In particular, a DCT is a Fourier-related transform similar to the discrete Fourier transform (DFT), but using only real numbers. DCTs are equivalent to DFTs of roughly twice the length, operating on real data with even symmetry (since the Fourier transform of a real and even function is real and even), where in some variants the input and output data are shifted by half a sample. There are eight standard DCT variants, of which four are common.

The most common variant of discrete cosine transform is the type-II DCT, which is often called simply “the DCT”; its inverse, the type-III DCT, is correspondingly often called simply “the inverse DCT” or “the IDCT”. Two related transforms are the discrete sine transform (DST), which is equivalent to a DFT of real and odd functions, and the modified discrete cosine transform (MDCT), which is based on a DCT of overlapping data.

The DCT, and in particular the DCT-II, is often used in signal and image processing, especially for lossy data compression, because it has a strong “energy compaction” property. Most of the signal information tends to be concentrated in a few low-frequency components of the DCT.

3. Neural Network

In this work one Neural Network structure is used, which is Learning Vector Quantization Neural Network. A brief overview of this network is given below.

3.1 Learning Vector Quantization

Learning Vector Quantisation (LVQ) is a supervised version of vector quantisation, similar to Selforganising Maps (SOM) based on work of Linde et al, Gray and Kohonen. It can be applied to pattern recognition, multi-class classification and data compression tasks, e.g. speech recognition, image processing or customer classification. As supervised method, LVQ uses known target output classifications for each input pattern of the form.

LVQ algorithms do not approximate density functions of class samples like Vector Quantisation or Probabilistic Neural Networks do, but directly define class boundaries based on prototypes, a nearest-neighbour rule and a winner-takes-it-all paradigm. The main idea is to cover the input space of samples with ‘codebook vectors’ (CVs), each representing a region labelled with a class. A CV can be seen as a prototype of a class member, localized in the centre of a class or decision region in the input space. A class can be represented by an arbitrarily number of CVs, but one CV represents one class only.

In terms of neural networks a LVQ is a feedforward net with one hidden layer of neurons, fully connected with the input layer. A CV can be seen as a hidden neuron (‘Kohonen neuron’) or a weight vector of the weights between all input neurons and the regarded Kohonen neuron respectively.

Learning means modifying the weights in accordance with adapting rules and, therefore, changing the position of a CV in the input space. Since class boundaries are built piecewise-linearly as segments of the mid-planes between CVs of neighbouring classes, the class boundaries are adjusted during the learning process. The tessellation induced by the set of CVs is optimal if all data within one cell indeed belong to the same class. Classification after learning is based on a presented sample’s vicinity to the CVs: the classifier assigns the same class label to all samples that fall into the same tessellation – the label of the cell

Reference

Brain plasticity

drios — Mon, 16 Jun 2008 11:00:00 +0000

Neuroscience has altered considerably in the earlier 20 years. An example of change over phase is the theory of brain plasticity. Brain plasticity refers to the brain’s ability to rewire itself, relocating information processing functions to different brain areas and/or neural networks. Two decades ago, it was assumed that brain networks were static after its early formation time. Now that belief has distorted. The analysis of brain plasticity has profound implications in human learning and behavior, and as such, for mental strength.

To better understand this idea, let’s take a short tour of the human brain, neural networks, and the plastic potential therein.

Brains, Neurons and Networks

The brain is a multilayered parallel structure in which billions of neurons are interconnected and swap information through neural networks. In the brain, each neuron is connected to thousands of other neurons through synapses (specialised neuronal junctions). A connected neuron receives input from numerous other neurons, and when the input weight reaches a activation value, the neuron propagates an electrical signal that stimulates output through the ignition of a neurotransmitter (input to another neuron).

This electrochemical trade is the core of brain cell communication. It is also the premise for the formation of neural networks. These networks are produced during early childhood and are responsible for particular brain tasks, such as learning, pattern recognition and problem-solving. It was said that once neural networks were formed, they would remain ‘hard-wired’ or inflexible. However, inquiries in the past two decades has indicated that this is not the reality: our neural networks are in fact adaptive, flexible and responsive to change.

Rewiring is the Key

So what does it really mean to have a plastic brain? It has many implications to human behaviour and learning patterns. Primarily, it defies the old adage that "an old dog cannot learn new tricks". It is clear that with age, it becomes increasingly more complex to learn new things. However, the brain’s ability to adapt to change perpetuates throughout an individual’s lifetime.

A prominent case of neuroplasticity happened with a patient who spent 19 years in a coma. Terry Wallis, a 19 year old man from Massachusetts (US), woke up after spending 19 years in a minimally conscious state. When scientists scanned his brain combining PET (Positron Emission Tomography) and DTI (Diffusion Tensor Imaging) technologies, they found data that Wallis’s brain had "developed new pathways and completely novel anatomical structures to re-establish functional connections, compensating for the brain pathways misplaced in the accident" (New Scientist, 03/07/2006).

Other cases, including stroke victims, people who have lost sensorial abilities (e.g. visually impaired) and individuals who have suffered cortical injuries show analogous conclusions after researchers have investigated how they have improved, or how the brain rewired itself to compensate for the injured areas and lost functions. The process of rewiring occurs when new connections (synapses) between neurons are formed and, if they prove to be favourable, they are likely to become more permanent and stabilised. This method allows the brain circuitry to be malleable to changes, or in other words, to form ‘uncommon’ networks under particular conditions.

Learning and Plasticity

Brain plasticity is not restricted to unplanned circumstances, such as accidents, brain traumas and other critical instances that require rewiring to re-establish functional connections. Learning is also a major beneficiary of brain plasticity. Studies with musicians and athletes have shown that particular areas of the brain responsible for ‘fine’ or ‘specific’ movements in certain parts of the body (e.g. the hands of a pianist or string musician) are in fact rewired for optimization. Once training becomes a routine, and particular actions are repeated over and over again, the tendency is that neuronal connections will become more permanent.

But there is more to it. Material contact is not a requirement when it comes to rewiring. Repeated thinking can also trigger a sequence of reactions which result in brain rewiring. Scientists have investigated the formation of synapses as a product of ‘thinking about doing something’ and found that, from a neuronal perspective, thinking can be as useful as doing. This evidence led to an interesting fusion of interests between Buddhist meditation (through the Dalai Lama’s interest on the influence of the mind over the brain) and the scientific study on brain plasticity and the formation of neural networks. It seems that brain plasticity is a flexible topic as well as a stretchy concept.

Mind Your Thoughts

Learning and plasticity took center stage when collaborative research was conducted with lamas (Buddhist equivalent for priests or spiritual leaders). It seems that, as a result of ongoing meditation through a technique called Mindfulness (which aims to develop the person’s control and awareness of thoughts and emotions), the lamas were ‘more able’ to attain emotional consider and to concentrate.

Some of these studies include experiments performed by Dr. Kabat-Zinn (who educated mindfulness to employees in a high-pressure biotech business and concluded that stress levels were optimized over a short period of time) and Dr. Ekman’s tests involving emotional expression detections. "The mindfulness training focuses on learning to monitor the continuing sensations and thoughts more closely, both in sitting meditation and in activities like yoga exercises" (NY Times, 04/02/2003).

The benefits of meditation through brain rewiring, from a non-religious perspective, are becoming clearer and quite appealing. Currently, there are therapeutic techniques that mix mindfulness with other mainstream therapies such as Cognitive Behavior Therapy. These have proven particularly useful for cases of depression and anxiety, for example.

Stepping Into the Strange

Brain plasticity has become a major topic of study. As fresh scanning technologies allow scientists to monitor the formation of synapses under particular stimuli, and experiment with living organisms, the applications of this knowledge are getting a range of research fields. Some scientists have promoted the idea of using stimulation to increase learning, however, at a neurochemical level. Others like the idea of meditation and ‘wishful thinking’ to empower the procedure of learning and to optimize the performance of certain tasks.

This collaborative approach from representatives of a non-dogmatic religion such as Buddhism, cognitive researchers and neuroscientists seems to be opening an attractive scope on the notion of brain plasticity. How far will this go? Hard to say, but nevertheless: very interesting to mind.

Perceptron neural network

drios — Fri, 23 May 2008 05:31:00 +0000

Perceptrons are the simplest architecture to learn when studying Neural Networking. Picture you mind of a perceptron as a node of a wide, interconnected neural network, sort of like a data tree, although the neural network does not necessarily have to have a top and bottom sections. The connections among all the nodes not only show the relationship between the nodes but also transmit data and information, called a signal or impulse. The perceptron is a simple model of a neuron .

Since making connections of perceptrons into a neural structure is a bit complicated, let’s take a perceptron by itself. A perceptron has a number of external input pattern, one internal input (called a bias), a threshhold, and one output. To the right, you can see a picture of a simple perceptron. It resembles a neuron.

Usually, the input values are boolean (just two possible values 1 and 0, true and false), but they can be any number. The output of the perceptron, however, is always boolean like a switch. When the output is on (has the value 1), the perceptron is said to be activated.

All of the inputs (including the bias) have weights attached to the input patterns that modify the input values to the neural network. The weight is just multiplied with the input.

The activation function is one of the key components of the perceptron as in the most common neural network architectures. It determines, based on the inputs, whether the perceptron activates or not. Basically, the perceptron takes all of the weighted input values and adds them together. If the sum is above or equal to some value (called the threshold) then the perceptron fires. Otherwise, the perceptron does not. So, it get active whenever the following equation is true (where w represents the weight, and there are n inputs):

The threshold is like a wall: if the “signal” has enough “energy” to jump over the wall, then it can keep going, but otherwise, it has to stop. Traditionally, the threshold value is represented either as the Greek letter theta (the symbol inside the circle in the picture above) or by a graphical symbol that looks like a square S:

The main feature of perceptrons is that they can be trained (or learn) to behave a certain way as all neural networks. One popular beginner’s assignment is to have a perceptron model (that is, learn to be) a basic boolean function such as AND or OR. Perceptron learning is supervised, that is, you have to have something that the perceptron can imitate. So, the perceptron learns like this: it produces an output, compares the output to what the output should be, and then adjusts itself a little bit. After repeating this cycle enough times, the perceptron will have converged (a technical name for learned) to the correct behavior.

This learning method is called the delta rule, because of the way the perceptron checks its accuracy. The difference between the perceptron’s output and the correct output is assigned the Greek letter delta, and the Weight i for Input i is altered like this (the i shows that the change is separate for each Weight, and each weight has its corresponding input):

Change in Weight i = Current Value of Input i × (Desired Output – Current Output)

This can be elegantly summed up to:

The delta rule works both if the perceptron’s output is too large and if it is too small. The new Weight i is found simply by adding the change for Weight i to the current value of Weight i.

Interestingly, if you graph the possible inputs on different axes of a mathematical graph, with pluses for where the perceptron fires and minuses where the perceptron doesn’t, the weights for the perceptron make up the equation of a line that separates the pluses and the minuses.

For instance, in the picture above, the pluses and minues represent the OR binary function. With a little bit of simple algebra, you can transform that equation in the diagram to the standard line form in which the weights can be seen clearly. (You get the following equation of the line if you take the firing equation and replace the “greater than or equal to” symbol with the equal sign).

This equation is significant, because single perceptron can only model functions whose graphical models are linearly separable. So, if there is no line (or plane, or hyperplane, etc. depending on the number of dimensions) that divides the fires and the non-fires (the pluses and minuses), then it isn’t possible for the perceptron to learn to behave with that pattern of firing. For instance, the boolean function XOR is not linearly separable, so you can’t model this boolean function with only one perceptron. The weight values just keep on shifting, and the perceptron never actually converges to one value.

So, by themselves, perceptrons are a bit limited, but that is their appeal. Perceptrons enable a pattern to be broken up into simpler parts that can each be modeled by a separate perceptron in a network. So, even though perceptrons are limited, they can be combined into one powerful network that can model a wide variety of patterns, such as XOR and many complex boolean expressions of more than one variable. These algorithms, however, are more complex in arrangement, and thus the learning function is slightly more complicated. For many problems (specifically, the linearly separable ones), a single perceptron will do, and the learning function for it is quite simple and easy to implement. The perceptron is an elegantly simple way to model a human neuron’s behavior. All you need is the first two equations shown above.

A detailed explanations about perceptron neural network can be found at Perceptron and Adaline Neural Networks

Backpropagation Neural network

drios — Thu, 08 May 2008 04:29:00 +0000

Introduction

Before we begin, know that our goal is to give you as much useful information as we can fit on our page.

This condition focuses on a particular print of neural network exemplar, known as a "supply-forwards back-propagation network". This exemplar is painless to understand, and can be openly employed as a software simulation.

First we will argue the foremost concepts behind this print of NN, then we’ll get into some of the more useful application dreams.

Complex problems

As we take the journey through the final part of this article, you can look back at the first part if you need any clarifications on what we have already learned.

The domain of neural netmachinery can be thought of as being allied to artificial intelligence, invent education, matching meansing, statistics, and other domains. The attraction of neural netmachinery is that they are best competent to solving the troubles that are the most strenuous to explain by traditional computational reasonings.

judge an aura meansing charge such as recognizing an everyday thing projected against a background of other things. This is a charge that even a small newborn’s wits can explain in a few tenths of a following. But house a conventional series invent to execute as well is incredibly dense. However, that same newborn might NOT be clever of calculating 2+2=4, while the series invent explains it in a few nanofollowings.

A fundamental difference between the aura recognition riddle and the addition riddle is that the past is best explaind in a matching frame, while easy mathematics is best done seriesly. Neurobiologists think that the wits is alike to a massively matching analog mainframe, containing about 10^10 easy meansors which each expect a few millifollowings to counter to store. With neural network technology, we can use matching meansing reasonings to explain some authentic-world troubles where it is very strenuous to classify a conventional algorithm.

The Feed-Forward Neural Network form

If we think the creature wits to be the ‘final’ neural network, then beliefly we would like to develop a invent which copys the wits’s performs. However, because of limits in our technology, we must mend for a greatly easyr invent. The evident line is to invent a small electronic invent which has a reassign perform alike to a biological neuron, and then link each neuron to many other neurons, with RLC netmachinery to copy the dendrites, axons, and synapses. This print of electronic exemplar is still instead dense to employ, and we may have strenuousy ‘thinking’ the network to do something effective. advance conslinets are wanted to make the invent more manageable. First, we change the linkivity between the neurons so that they are in patent layers, such that each neruon in one layer is linked to every neuron in the next layer. advance, we classify that suggests issue only in one focus across the network, and we simplify the neuron and synapse invent to perform as analog comparators being motivated by the other neurons through easy resistors. We now have a supply-forwards neural network exemplar that may actually be useful to develop and use.

Referring to numbers 1 and 2, the network performs as follows: Each neuron receives a suggest from the neurons in the preceding layer, and each of those suggests is multiplied by a part influence cost. The influenceed stores are summed, and agreed through a warning perform which scales the harvest to a preset stretch of costs. The harvest of the limiter is then advertise to all of the neurons in the next layer. So, to use the network to explain a riddle, we affect the store costs to the stores of the first layer, allocate the suggests to breed through the network, and read the harvest costs.

while the authentic uniqueness or ‘intelligence’ of the network exists in the costs of the influences between neurons, we essential a reasoning of adjusting the influences to explain a particular riddle. For this print of network, the most frequent education algorithm is called Back Propagation (BP). A BP network learns by example, that is, we must grant a education set that consists of some store examples and the known-rectify harvest for each container. So, we use these store-harvest examples to show the network what print of actions is likely, and the BP algorithm allocates the network to adapt.

The BP education means machinery in small iterative steps: one of the example containers is useful to the network, and the network produces some harvest based on the stream majesty of it’s synaptic influences (primarily, the harvest will be haphazard). This harvest is compared to the known-good harvest, and a mean-squared mistake suggest is calculated. The mistake cost is then breedd backwards through the network, and small changes are made to the influences in each layer. The influence changes are calculated to lessen the mistake suggest for the container in problem. The full means is recurring for each of the example containers, then back to the first container again, and so on. The round is recurring awaiting the general mistake cost drops below some pre-determined threshold. At this item we say that the network has erudite the riddle "well enough" – the network will never rigidly learn the belief perform, but instead it will asymptotically line the belief perform.

When to use (or not!) a BP Neural Network result

A back-propagation neural network is only useful in certain situations. next are some guidelines on when you should use another line:

* Can you write down a issue chart or a formula that accurately describes the riddle? If so, then spike with a traditional programming reasoning.

* Is there a easy instance of hardware or software that already does what you want? If so, then the development time for a NN might not be value it.

* Do you want the performality to "evolve" in a focus that is not pre-classifyd? If so, then think with a Genetic Algorithm (that’s another subject!).

* Do you have an painless way to cause a significant number of store/harvest examples of the beloved actions? If not, then you won’t be able to line your NN to do something.

* Is the riddle is very "discrete"? Can the rectify answer can be found in a look-up graph of reasonable mass? A look-up graph is greatly easyr and more accurate.

* Are rigid numeric harvest costs expectd? NN’s are not good at generous rigid numeric answers.

Conversely, here are some situations where a BP NN might be a good idea:

* A large total of store/harvest figures is existing, but you’re not loyal how to concern it to the harvest.

* The riddle appears to have overwhelming denseity, but there is openly a blend.

* It is painless to invent a number of examples of the rectify actions.

* The blend to the riddle may change over time, inside the bounds of the given store and harvest parameters (i.e., nowadays 2+2=4, but in the outlook we may find that 2+2=3.8).

* Outputs can be "fuzzy", or non-numeric.

One of the most frequent applications of NNs is in aura meansing. Some examples would be: identifying hand-printed characters; matching a photograph of a part’s face with a different photo in a figuresbase; executeing figures compression on an aura with token harm of content. Other applications could be: declare recognition; RADAR signature testing; cattle souk prediction. All of these troubles implicate large totals of figures, and dense relationships between the different parameters.

It is important to recall that with a NN blend, you do not have to understand the blend at all! This is a foremost gain of NN linees. With more traditional techniques, you must understand the stores, and the algorithms, and the harvests in great describe, to have any prospect of employing something that machinery. With a NN, you merely show it: "this is the rectify harvest, given this store". With an adequate total of lineing, the network will mimic the perform that you are demonstrating. advance, with a NN, it is OK to affect some stores that shot out to be irrelevant to the blend – during the lineing means, the network will learn to snub any stores that don’t contribute to the harvest. Conversely, if you authority out some vital stores, then you will find out because the network will bomb to meet on a blend.