Paths Towards Intelligence
Jan 01 2025
The grandest problem for humanity to solve is to understand and build intelligence. The brain is the most complex object known to humanity, and with general intelligence we can conceivably solve all (solvable) problems in the universe given enough time and resources.
Working towards understanding intelligence is a terminal goal of mine. But before really working on this, one first has to: define the problem and solution, and understand the different paths towards tackling this grand challenge. The latter will be discussed here, but I will first briefly explore what we are trying to understand and what will it look like when we do.
Defining the problem and solution
There are various definitions of intelligence, with no real consensus on the best way to formally define it. The fuzzy definition of general intelligence is whatever humans have, this is somewhat the idea behind the Turing Test - Assuming all knowledge can be represented through language, then if we cannot discern the difference between a human and AI; they must be equal. A more concrete definition is the ability to learn from experiences, adapt to new situations, plan and problem solve to manipulate the environment to achieve some goal.
It's also important to note that intelligence is on a spectrum, both humans and cats are intelligent beings, but humans are higher on the ladder. Ultimately the level of human intelligence and beyond is what we are trying to achieve here. Whilst understanding more basic forms of intelligence first will be useful, we will limit the discussion to the understanding and development of human level intelligence.
Next we need to know what it means to "understand" intelligence? In order to understand an intelligent system we should ultimately be able to make predictions on how it will behave given an environment and goals. We should also, in some capacity, be able to replicate an intelligent system. This isn't a prerequisite to understanding since the resources required could be inaccessible for current technology (the brain could be computationally irreducible). However, assuming there is nothing special going on in the brain besides raw computation, then there is nothing stopping a universal Turing machine from replicating it.
Paths
In my mind, there are five general paths towards understanding and building intelligence. All will likely work, but may converge on different solutions. Some paths are likely much more challenging than others, and which path is taken will depend on various factors, including what your instrumental goals are towards understanding intelligence (like making money, advancing biology, etc).
Replicating the brain
The most direct path is the pure neuroscience approach of completely reverse engineering the brain and creating a computational simulation of it. Fields like cognitive and computational neuroscience are already exploring the fundamental mechanisms of how the brain functions and how they can be modelled. There exist various mathematical models of neurons such as the Hodgkin-Huxley model, integrate-and-fire models, cable theory, etc. Which are laying the groundwork for modelling and understanding biological neurons.
The primary appeal of this path is its fidelity. Perfectly simulating the human brain would result in a system capable of any task a human can perform. Developing such a model would also advance computational biology and lead to breakthroughs in treating neurological disorders which is a bonus.
However, this approach has some clear caveats, mapping the brain at a cellular level is expensive and complex. Even the relatively simple fly connectome has only recently been mapped, and it lacks dynamic interaction details. Computationally, simulating biologically accurate neurons is far less efficient than current artificial neural networks. Specialized hardware like ASICs or projects like SpiNNaker might address these inefficiencies, but the gap remains significant.
Neuroscience Inspired Approach
Instead of fully replicating the brain, we could instead draw high-level inspiration from neuroscience. This is the approach of early machine learning models. With work such as perceptrons, Hebbian learning, Boltzman machines, reinforcement learning, spiking neural networks. With various new libraries being released that implement neuro-inspired algorithms.
This approach strikes a balance between being grounded in neuroscience and the existence proof of intelligence that is the brain, and computational efficiency since we don't require biological accuracy. This path can also be iterative, starting with basic high-level models and progressively incorporating more neuroscientific details as required.
One challenge is deciding how deeply to explore the brain’s mechanisms. Modern neural networks are based on the idea that "neurons that fire together, wire together". However, this is a very high-level model and is extremely simplified, missing a lot of key mechanisms of neurons. One example of this is that individual neurons can learn to solve the XOR problem, yet artificial perceptions famously cannot. Whilst this approach does not require atom level precision, it seems further biological inspirations may still be required.
Why isn’t this approach more popular? Primarily because neuroscience-inspired models often underperform compared to state-of-the-art machine learning systems on current ML benchmarks or are designed to address fundamentally different objectives. Existing datasets might not align with the kind of modular, abstract components this approach might produce. For example, could the human hippocampus alone solve ImageNet? Likely not, though it’s a crucial component of intelligence.
Machine Learning Approach
Back in the 80's, machine learning was an endeavour to create computational models of the brain, applying the approach described above. However, these days the field of machine learning has transitioned to developing models that perform well on datasets and benchmarks, and slowly iterating on these models to gain minor improvements. Without a whole lot of focus on basing these algorithms on the brain.
Methods such as diffusion models don't have much basis in biology, but they work super well for generating high-quality images from textual prompts. A more general and wildly popular architecture, the transformer, also doesn't have a huge grounding in biology. Instead of moving these architectures towards more neuroscience inspired methods, researchers are focusing more on purely computational improvements, such as better positional encoders to optimise performance. Due top this, I consider this to be a separate approach to the one above.
This approach has the benefit of high training and inference speeds, and immediate value in real-world applications. Over the last decade we have made huge improvements in the field, recent LLMs have displayed remarkable performance despite their simplicity in design.
Whilst this lacks biological accuracy, it isn't necessarily a issue since there are likely various ways to create intelligence. However, the further away from the human brain we get, the more we are relying on small incremental improvements to get us there. And if our current approaches are leading us down the wrong path this could result in some major setbacks, all for some incremental improvements in some benchmark.
Evolutionary approach
This approach takes a step back from the neuroscience approach. Here, we are still replicating nature, but not the brain, instead replicating the algorithm used to create the brain, evolution. Since evolution took us from single-cell organisms to general purpose intelligence machines, why not just replicate that process? This is plausible in theory, but simulating millions of agents over vast timescales is computationally infeasible. However, we can still take inspiration from evolution.
We could potentially apply evolutionary algorithms to the problem of developing structures of neurons that can exhibit complex behaviour. This would still be challenging, but networks such as NEAT already exist which evolve neural network topologies. Due to the gene bottleneck, evolution learned to create the brain via repeating structures such as cortical columns using a few different neuron types. It is possible that evolutionary algorithms, under similar constraints could develop different neuron types and architectures which could act as a fundamental unit of computation, repeated millions of times to form emergent intelligence.
Whilst this distillation of the problem greatly reduces our search space and makes evolutionary approaches potentially feasible. There is still a large search space to cover. Whilst evolutionary algorithms are guided by fitness functions, there is still a lot of stochasticity, with little guidance provided by humans or neuroscience.
Symbolic Approach
Popular in the 1950's, this approach involves creating systems based on symbols, axioms, and logical rules. While it offers transparency and interpretability, its rigidity and lack of adaptability have produced limited success thus far. Hybrid approaches such as AlphaGeometry combine symbolic reasoning with neural networks show that symbolic reasoning does have a place in this exploration when used in conjunction with connectionist ideas.
Whilst symbolic approaches will likely play a role in fields such as mathematics, via machine assisted proofs, I don't see it being robust enough to achieve generality. This is due to the huge amount of axioms required to embed the system with knowledge, which still can be brittle. The lacking ability of symbolic agents to learn new axioms from its environment also greatly restrict its ability to continually learn and develop its intelligence.
Choosing a Path
In the path towards intelligence then, which direction is the most promising? Hopefully, they are all explored to some extent, the human race acts as a nice parallelised search algorithm for ideas. But for an individual or small group, refined focus is more essential.
Over the months of exploring these paths and speculating on the optimal choice, my belief is that we need to take inspiration from neuroscience, but only look as deep as required. Initially understanding the brain in high-level concepts, and iteratively exploring lower levels until understanding is achieved. This gives us the benefit of staying grounded to the only existence proof of general intelligence we know, whilst being able to explore more computationally efficient and interpretable implementations.
Current machine learning research seems to be pointing in the direction of a pure machine learning approach. Not exploring insights from neuroscience, but optimising what we already have. With the advent of the transformer architecture, being a highly capable and general system, many researchers seem reluctant to stray down a different path. It seems difficult to go back to the days in which models perform poorly, but that may be required to explore completely different (but potentially more fruitful) paths. It's important to note that current machine learning systems are amazing, and produce incredible value in various ways, but they may not lead to understanding a general intelligent system. There is also a bit of a divide between the fields of machine learning and neuroscience, an interdisciplinary collaboration with great potential.
There are many issues with understanding general intelligence as of currently. Modern neural networks, whilst impressive, are hard to understand. Even if they did lead to AGI, we may have about as much understanding of them as we do our own brains (though it would be much easier to probe them). There are also issues with benchmarks. Current machine learning datasets are quite limited and don't represent general intelligence. I believe new benchmarks are required, both to test for general intelligence, and to explore the intermediate goals (such as simulating certain cortical regions).
I look forward to the explorations of these paths over the coming years to bring humanity one step closer to the ultimate achievement.