Every example neural network for image recognition I’ve read about produces a simple “yes” or “no” answer. One exit node corresponds to “Yes, this is a human face,” and one corresponds to “No, this is not a human face.”
I understand that this is likely for simplicity of explanation, but I’m wondering how such a neural network could be programmed to give a more specific output. For example, let’s say I was classifying animals. Instead of it saying “Animal” or “Not an animal”, I would want responses like “Dog”, “Fish”, “Bird”, “Snake”, etc., with one final exit node being “Not an animal/I don’t recognize this”.
I’m sure this must be possible, but I’m having trouble understanding how. It seems like due to the training algorithm of backpropogation of error, as you train up one exit node (i.e., “This is a dog”) and the weights of the neurons are changed, then the ideal state for another exit node that you previously trained (i.e., “This is a bird”) will begin to deviate, and vice versa. So training the network to recognize one category would sabotage any training done for another category, thus limiting us to a simple “Yes” or “No” design.
Does this make such a recognizer impossible? Or am I misunderstanding the algorithm? The only two things I can think of are that:
-
Either we could train one neural network for each thing we want classified and somehow use those to construct a greater, super-network (so for example, a network for “dog”, a network for “bird”, etc., which we somehow add together to create the super-network for “animals”); or,
-
Create some kind of ridiculously complicated training methodology which would require incredibly advanced mathematics and would somehow produce an ideal neuron-weight-state for all possible outputs (in other words,
insert math magic here).
(Side note 1: I am specifically looking at multilayer perceptrons as a kind of neural network.)
(Side note 2: For the first bulleted “possible solution”, having each specific neural network and iterating through them until we receive a “Yes” response is not good enough. I know this could be done fairly easily, but that is simple functional programming rather than machine learning. I want to know if it’s possible to have one neural network to feed the information to and receive the appropriate response.)
To answer just your title, yes. Neural nets can give non-boolean answers. For example, neural nets have been used to predict stock market values, which is a numeric answer and thus more than just yes/no. Neural nets are also used in handwriting recognition, in which the output can be one of a whole range of characters – the whole alphabet, the numbers, and punctuation.
To focus more on your example – recognising animals – I’d say it’s possible. It’s mostly an extension of the handwriting recognition example; you’re recognising features of a shape and comparing them to “ideal” shapes to see which matches. The issues are technical, rather than theoretical. Handwriting, when run through recognition software, is usually mapped down to a set of lines and curves – nice and simple. Animal faces are harder to recognise, so you’d need image processing logic to extract features like eyes, nose, mouth, rough skull outline etc. Still, you only asked if it’s possible, not how, so the answer is yes.
Your best bet is probably to take a look at things like Adaptive Resonance Theory. The general principle is that the sensory input (in this case, metrics on the relative size, shape and spacing of the various facial features) is compared to a “prototype” or template which defines that class of thing. If the difference between the sensory input and the remembered template is below a certain threshold (as defined by a “vigilance parameter”), then the object being observed is assumed to be a member of the group represented by the template; if no match can be found then the system declares it to be a previously unseen type. The nice thing about this sort of net is that when it recognises that an object is, say, a horse, it can learn more about recognising horses so that it can tell the difference between, say, a standing horse and a sleeping horse, but when it sees something new, it can start learning about the new thing until it can say “I don’t know what this is, but I know it’s the same thing as this other thing I saw previously”.
EDIT:
(In the interest of full disclosure: I’m still researching this myself for a project, so my knowledge is still incomplete and possibly a little off in places.)
how does this tie in with backpropogation setting weights for one output node ruining the weights for another, previously-trained node?
From what I’ve read so far, the ART paradigm is slightly different; it’s split into two sections – one that learns the inputs, and one that learns the outputs for them. This means that when it comes across an input set that doesn’t match, an uncommitted neuron is activated and adjusted to match the input, so that that neuron will trigger a match next time. The neurons in this layer are only for recognition. Once this layer finds a match, the inputs are handed to the layer beneath, which is the one that calculates the response. For your situation, this layer would likely be very simple. The system I’m looking at is learning to drive. This is actually two types of learning; one is learning to drive in a variety of situations, and the other is learning to recognise the situation. For example, you have to learn how to drive on a slippery road, but you also have to learn to feel when the road you’re driving on is slippery.
This idea of learning new inputs without ruining previously-learned behaviours is known as the stability/plasticity dilemma. A net needs to be stable enough to keep learned behaviour, but plastic enough that it can be taught new things when circumstances change. This is exactly what ART nets are intended to solve.
2
@anaximander’s answer is pretty good, I thought I’d comment on this part of your question:
It seems like due to the training algorithm of backpropogation of error, as you train up one exit node (i.e., “This is a dog”) and the weights of the neurons are changed, then the ideal state for another exit node that you previously trained (i.e., “This is a bird”) will begin to deviate, and vice versa. So training the network to recognize one category would sabotage any training done for another category, thus limiting us to a simple “Yes” or “No” design.
Well, I think your assumption is wrong here; if I understand correctly, you have an NN with one output per category you’re trying to classify. Ideally, you’d like them to operate almost independently, so that the “dog” and “bird” classification don’t fire at the same time. So really, during training, what will happen is that when you are training the NN with a “dog” result, backpropagation will try to ensure that the “bird” and other output neurons don’t produce false positives. So, theoretically, it will work just fine, contrary to your comment; the reinforcement of a negative outcome for “bird” is correct.
However, your problem will be the scalability of this approach. As you add more categories to the network, the training will become more complex in at least a linear (but likely far worse) manner. For this reason, many people use an approach where individual NNs are trained for each category; this keeps things simple enough and relatively scalable. The meta-level of how these are combined are then up to you. You could simply loop through all NNs and see which ones produce positive outputs, you could create mid-level heuristic NNs which try to narrow down the type of animal for you, or you could even have a giant NN which combines the individual NNs as neurons. In essence, what I am trying to say is that you have prior knowledge of the structure of the problem – the individual classifications are most likely distinct from each other; you can let a giant NN figure that out for itself (but it might not work well with today’s technology), or you can architect your NNs by hand to reflect this, which may yield better results.
EDIT: To answer the title question, of course NNs can provide more than yes / no answers. In the “standard” models, each output neuron typically fires yes / no (although this behaviour could be changed, if you were so inclined), representing one bit of information; but just as with your computer, bits can be combined to provide a range of discrete values, which can be interpreted any way you wish to. A fairly visual example of non-binary outputs would be a Self-Organising Map, which typically have a 2D output.
1
Short and not very rigid answer: yes every NN can provide more info than just yes, or no. It is due to threshold. If weights are higher than some threshold the answer is one of the classification classes, if it is lower the answer is the second classification class. Basically:
0..threshold
threshold..1
Neuron’s output is in the interval [0..1] (or [-1,1] it depends), and you don’t want to get answer whether the output is lower or higher than the threshold but the output (the output you can easily convert into 0..1 interval and that means %)