Neuromorphic chips like Intel Loihi 2 process spiking neural networks with orders-of-magnitude better energy efficiency than GPUs, but they cannot run standard backpropagation. The training challenge is fundamental: spikes are discrete binary events with no gradient, which breaks the continuous calculus that backprop depends on. In 2026, the field is converging on training SNNs on GPUs using surrogate gradients and then deploying the trained weights to neuromorphic hardware.
Pithy Cyborg | AI FAQs – The Details
Question: How are neuromorphic computing architectures (e.g., Intel Loihi 2) addressing spiking neural network training challenges compared to GPU-based backprop?
Asked by: Claude Sonnet 4.6
Answered by: Mike D (MrComputerScience)
From Pithy Cyborg | AI News Made Simple
And Pithy Security | Cybersecurity News
Why Standard Backpropagation Cannot Train Spiking Neural Networks Directly
Standard artificial neural networks use continuous activation functions like ReLU or GELU. Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule across these continuous functions. The math requires that every activation function has a well-defined derivative almost everywhere.
Spiking neural networks replace continuous activations with binary spike events: a neuron either fires (1) or does not fire (0) at each timestep. Spikes model biological neurons far more accurately than continuous activations and enable event-driven computation, where processing only occurs when a spike arrives rather than at every clock cycle. This is why neuromorphic hardware is so energy-efficient: most neurons are silent most of the time, and the hardware only consumes power when spikes propagate.
The derivative of a step function is zero almost everywhere and undefined at the threshold. Pass a zero gradient backward through ten layers and every weight receives a zero update. Training stalls completely. This is not a hardware limitation or an implementation detail. It is a mathematical incompatibility between discrete spike events and gradient-based optimization.
Early attempts to train SNNs used spike timing-dependent plasticity (STDP), a biologically inspired local learning rule that adjusts synaptic weights based on the relative timing of pre- and post-synaptic spikes. STDP can train shallow networks on simple tasks but scales poorly to deep architectures and does not converge on complex recognition tasks that deep learning handles easily. The gap in task performance between STDP-trained SNNs and backprop-trained ANNs remained embarrassingly large for most of the 2010s.
How Surrogate Gradients Bridge the Training Gap
The surrogate gradient method, which became the dominant SNN training approach around 2018 and remains so in 2026, solves the discontinuous derivative problem with a pragmatic approximation.
During the forward pass, the network uses real spike events: binary 0 or 1 at each timestep. During the backward pass, the derivative of the spike function is replaced with a smooth surrogate, typically a sigmoid or piecewise linear function centered at the firing threshold. The surrogate has a well-defined gradient everywhere, so backpropagation proceeds normally. The weights are updated based on gradients computed through this surrogate, not through the true spike function.
The result is that the network learns to produce useful spike patterns even though the gradients during training are technically computing derivatives of a function that is not the one being used in the forward pass. Empirically this works remarkably well. SNNs trained with surrogate gradients on GPU now approach within a few percentage points of equivalent ANN performance on image classification benchmarks, and outperform ANNs on certain temporal pattern recognition tasks where the spike timing carries information.
Training still happens on GPUs. Intel Loihi 2 and IBM’s NorthPole do not run backpropagation in any form. The workflow is train on GPU using surrogate gradients in PyTorch or the SpikingJelly framework, then convert the trained SNN weights to the neuromorphic chip’s format for deployment. AI hardware acceleration benchmarks consistently show neuromorphic chips delivering 100x to 1000x better energy efficiency than GPUs for SNN inference on the right workloads, which is the reason the field continues investing in the training complexity.
What Intel Loihi 2 and NorthPole Actually Offer in 2026
Intel’s Loihi 2, released in 2021, contains 1 million neurons and 120 million synapses implemented in Intel 4 process technology. It is not a general-purpose chip. It executes spike-based computation natively, with each neurocore handling thousands of neurons in parallel using event-driven processing that consumes power only when spikes are present.
The energy numbers are striking for the right workloads. Running a keyword spotting SNN on Loihi 2 consumes roughly 1 milliwatt. Running an equivalent ANN on a GPU for the same task consumes hundreds of milliwatts to several watts. For always-on edge sensing applications, battery-operated IoT devices, and implantable medical devices, this difference is the difference between a device that runs for years on a coin cell and one that requires frequent recharging.
IBM’s NorthPole, announced in late 2023, takes a different approach. Rather than pure neuromorphic design, NorthPole implements dense on-chip memory co-located with compute to eliminate off-chip memory access almost entirely during inference. It achieves 22 tera-operations per second per watt on ResNet-50, dramatically outperforming GPU efficiency for standard ANN inference. NorthPole is not strictly neuromorphic but represents the same architectural philosophy: compute where the data lives, fire only when needed.
The honest limitation in 2026 is that neither Loihi 2 nor NorthPole nor any neuromorphic chip runs large language models efficiently. The workloads where neuromorphic architecture wins are sparse, temporally structured, and relatively shallow: sensory processing, anomaly detection, keyword spotting, and event-based vision from neuromorphic cameras. The dense matrix operations that dominate transformer inference remain GPU and TPU territory.
What This Means For You
- Use surrogate gradient training in SpikingJelly or Norse (PyTorch-based SNN frameworks) if you are experimenting with SNNs. These implement the full training pipeline on GPU and handle the forward-backward pass mismatch transparently.
- Target neuromorphic deployment for always-on edge workloads where energy efficiency dominates. Keyword spotting, vibration anomaly detection, and event-based vision are the sweet spots where Loihi 2 delivers its energy advantage over GPU inference.
- Do not expect neuromorphic chips to replace GPUs for LLM inference in the near term. The architectural mismatch between dense transformer operations and sparse spike-based processing is fundamental, not a gap that more chip iterations will close.
- Watch IBM NorthPole’s ANN inference efficiency as a bellwether for near-memory computing applied to standard deep learning. It represents a middle path between neuromorphic and conventional GPU architectures.
- Evaluate Intel’s Loihi research cloud access program if you want to prototype on actual neuromorphic hardware without acquiring chips directly. Intel provides cloud access to Loihi 2 for academic and research projects.
Pithy Cyborg | AI News Made Simple
Subscribe (Free): https://pithycyborg.substack.com/subscribe
Read archives (Free): https://pithycyborg.substack.com/archive
Pithy Security | Cybersecurity News
Subscribe (Free): https://pithysecurity.substack.com/subscribe
Read archives (Free): https://pithysecurity.substack.com/archive