Beyond Depth and Width: Why "Taller" Neural Networks are the Next Leap in Model Efficiency

When we look back at the foundational principles of the deep learning revolution, one clear architectural truth defined an era: deeper networks beat wider networks.

Very early neural network designs often focused on adding units to widen a shallow network. However, adding depth (adding layers sequentially) unlocked vastly superior capacity. Each additional layer allowed the network to build increasingly more sophisticated representations by transforming the outputs of the previous layer. This hierarchical representation learning is what allowed deep learning models to tackle complex tasks that were previously far out of reach, such as object recognition in images and speech recognition.

Today, standard neural network architectures still largely rely on these two fundamental dimensions: width and depth.

At Perforated, we are introducing a third dimension: height.

The Third Dimension: Building “Taller” Networks

Perforated training can be conceptualized as moving beyond the traditional 2D grid of depth and width. By introducing a third dimension, we position a new class of units (which we call dendrites) at a level “above” conventional network units.

Here is how height changes the equation:

Targeted Support: Dendrites are placed above conventional units and are separately trained to observe and provide feedback on the performance of the units located below them.

Stackable Capacity: These dendrites can be stacked higher and higher, creating a vertical hierarchy that delivers compounding performance improvements to the primary units at the lowest level.

This training approach allows us to train small models that rival, or even surpass, the performance of much larger models.

How Perforated Training Operates Under the Hood

Rather than training a model with a fixed architecture and parameter count, we strategically grow the neural network throughout the training process. By training new units differently than the rest of the network, and only adding them when necessary, we ensure those parameters boost network performance much more than ‘conventional’ units.

1. Primary units in the network undergo conventional training.
2. Concurrently, new units (dendrites) are trained separately to directly optimize and support existing units, systematically reducing their average training error.
3. Once trained, these supporting units are integrated into the network, improving representation quality for the units they assist.

Just as deep learning today relies on an ecosystem of non-trivial mechanics (optimizers, learning rate schedules, normalization layers, regularization, and gradient diagnostic tools), Perforated training encompasses a broad suite of engineering approaches to ensure training performance and efficiency.

Solving a 40-Year Scaling Challenge

Attempting to dynamically add parameters during training isn’t new. Computer scientists have experimented with constructive neural network algorithms for decades. However, many of these algorithms have struggled to scale to modern deep learning workloads.

Perforated is the first paradigm to make constructive training scale to today’s massive models. By structuring parameter addition along the dimension of height, we have established a new class of data-efficient training.

We have demonstrated these results in real-world environments:

• Proven Scalability: Successfully applied across production customer models.
• Architectural Versatility: Validated across multiple distinct model architectures.
• Enterprise-Scale Execution: Proven performance on enterprise-scale models.

By shifting the focus from simply expanding model width or depth to actively improving existing feature representations, we are unlocking significant improvements in data efficiency and model capacity.

---