Backpropagation enabled neural network optimization, but is still expensive at scale. An efficient alternative for neural networks must reduce scaling costs and provide high-efficiency optimizations. The presented solution for feed-forward
layers is extended compositionally, for transformers containing feed-forward and self-attention
layers. These
train complex models modularized by self-attentive feed-forward units (SAFFUs), defining efficient architectures that generalize over much less data. Results demonstrate outperformance of
backpropagation (alone), compared to application of
backpropagation after explicit solutions, which uncover better optima from less data. Ablations
train a roadmap of 245 transformers to determine specifications for the SAFFU-
transformer, showing several architectural variants appear quite performant. The most performant models appear to not be the most parameterized. Summarizing, well-generalized SAFFU models are efficiently reachable, their architectural explorability via explicit solutions is cheaper and more robust than by
backpropagation, and their explicit solutions could be useful for low-resource hardware, where AI might be embodied.