Algorithm to remove orphan neurons from a neural network - algorithm

Question

I'm trying to implement NEAT (Neuro Evolution of Augmenting Topologies).
I have a list of network connections, called "genes". A connection between neuron1 and neuron2 would be gene.from = neuron1, gene.to = neuron2.
My task is to generate a neural network from these genes (The neural network is simply a map from index to neuron, the gene.from and gene.to are the keys to the neurons in the map).
I have numPossibleInputs input nodes, so we add those first (0-numPossibleInputs-1 are input neurons).
I have numOutputs output nodes, so we add those as well.
Then, we sort our genes based on their "to" connection indices.
Finally, we create the hidden layer neurons based on the genes... As the neural network is a map, we just check if the to or from of a connection is already a neuron, otherwise create a new one. This algorithm creates networks just fine.
public void generateNetwork()
{
neuralNetwork.clear();
for(int i = 0; i < numPossibleInputs; i++)
{
neuralNetwork.put(i, new Neuron());
}
for(int i = 0; i < numOutputs; i++)
{
neuralNetwork.put(i+numPossibleInputs+numPossibleHidden, new Neuron());
}
genes.sort((ConnectionGene g1, ConnectionGene g2)-> Integer.compare(g1.toNeuronIndex, g2.toNeuronIndex));
for(ConnectionGene gene : getCleanGenes(genes))
{
if(gene.enabled)
{
if(!neuralNetwork.containsKey(gene.toNeuronIndex))
{
neuralNetwork.put(gene.toNeuronIndex, new Neuron());
}
neuralNetwork.get(gene.toNeuronIndex).incomingConnections.add(gene); // Add this gene to the incoming of the above neuron
if(!neuralNetwork.containsKey(gene.fromNeuronIndex))
{
neuralNetwork.put(gene.fromNeuronIndex, new Neuron());
}
}
}
}
The problem comes when the evolution algorithm turns "off" some of the genes (note the gene.enabled). For example, consider the following genes (There are others, but they are disabled):
2->4
4->4
13->4
0->13
1->13
5->13
We also have disabled genes, 2->5 and 4->13. These can not be used in the network as they arent being expressed. (This is why i have to generate a new network every generation, genes can be added, enabled, disabled, etc.).
This is for numPossibleInputs ==3, so 0 1 and 2 are inputs (2 is bias). 5 is a hidden layer node since 5 > 3, but less than 10 + 3 = 13. 13 is an output node, i had numPossibleHidden == 10 so 10 + 3 = 13... just one output.
Can picture it like this:
[input input input hidden*10 output*1] for 3 inputs, 10 hidden, and 1 output
This is a picture of that network naively generated:
Simple Network
As we can see, the reduced network shouldn't have 4 or 5 at all, since they have no influence on any outputs (In this case only one output, 13). The reduced neural network would just be 0->13 and 1->13.
I had some initial thoughts on how to solve this:
A.
1. Loop over each connection and hash the gene.from ids. These are the neuron ids which are an input to something else
2. After populating the hash, loop again and remove any genes with gene.to not being in the hash (The gene.to is not an input to anything else if it isnt in the hash).
3. Repeat until we don't remove anything
B. Generate the naive network... then, Crawl backwards in the network, from each output until we can't go any further (take care for recurring cycles). Hash each node we find. Once our graph search is done, compare our hash of nodes found with the total nodes expressed in our gene list. Only use genes with neurons in the hash of found nodes and remake the network.
I was hoping to get some feedback on what might be the best algorithm to do this based on my network representation - I'm thinking my B is better than A, but I was hoping there was a more elegant solution that didn't involve me parsing graph topology. Perhaps something clever I can do by sorting the connections (By to, by from)?
Thanks!

Answer 1

I used my B solution above, tested it with all kinds of different network typologies, and it works fine - That is, the network will get rid of all nodes who do not have a proper path from inputs to outputs. I'll post the code here in case anyone wants to use it:
private List<ConnectionGene> cleanGenes(Map<Integer,Neuron> network)
{
// For each output, go backwards
Set<Integer> visited = new HashSet();
for(int i = 0; i < numOutputs; i++)
{
visited.add(i+numPossibleInputs+numPossibleHidden);
cleanGenes(i+numPossibleInputs+numPossibleHidden, network, visited);
}
List<ConnectionGene> slimGenes = new ArrayList();
for(ConnectionGene gene : genes)
{
// Only keep gene if from/to are nodes we visited
if(visited.contains(gene.fromNeuronIndex) && visited.contains(gene.toNeuronIndex))
{
slimGenes.add(gene);
}
}
return slimGenes;
}
private boolean cleanGenes(int neuronIndex, Map<Integer, Neuron> network, Set<Integer> visited)
{
int numGoodConnections = 0;
for(ConnectionGene gene : network.get(neuronIndex).incomingConnections)
{
numGoodConnections++;
if(gene.enabled && !visited.contains(gene.fromNeuronIndex))
{
visited.add(gene.fromNeuronIndex);
if(!cleanGenes(gene.fromNeuronIndex, network, visited))
{
numGoodConnections--;
visited.remove(gene.fromNeuronIndex); // We don't want this node in visited, it has no incoming connections and isn't an input.
}
}
}
if(numGoodConnections == 0)
{
return neuronIndex < numPossibleInputs; // True if an input neuron, false if no incoming connections and not input
}
return true; // Success
}
According to my profiler, the vast majority of time spent in this NEAT algorithm is in the simulation itself. That is, generating the proper network is trivial compared to testing the network against a hard task.

Answer 2

There is a much more efficient way to add neurons. Instead of just adding a new neuron and hopeing for it to be connected one day, you could also take a random connection, split it up in two connections and add a neuron between them.