Someone fitted neural textures using evolution strategies instead of backprop

Rich Geldreich wrote up an experiment fitting neural textures to PBR material maps using evolution strategies instead of backpropagation. The approach treats the neural network as a black box. Mutate the weights, measure how close the output is to the target texture, keep the mutations that work. This is interesting because fitting neural textures is normally a gradient descent problem. You run backprop through a tiny MLP that takes UV coordinates and outputs colour or roughness. Evolution strategies skip that entirely. No autodiff framework, no learning rate tuning, just a population of candidate networks competing to approximate the material. The post does not claim ES is faster or better. It reads more like curiosity about whether you can dodge the backprop machinery entirely and still converge. The answer seems to be yes, but slowly. ES scales badly with parameter count because every fitness evaluation is a full forward pass, and you need thousands of evaluations per generation. What caught my attention is the question of when you would actually want this. If you are running on hardware without good autodiff support, or if your loss function is not differentiable, ES starts to make sense. Texture fitting is smooth enough that gradients work fine, so this is more proof-of-concept than practical. Still, the idea of treating a neural network as an opaque function you can optimise with search instead of calculus is appealing. Especially in domains where the loss surface is full of discontinuities or where you need to optimise discrete choices alongside continuous parameters. Evolution strategies handle that without special casing. I would be curious to see this approach applied to something like procedural texture generation where the target is not a fixed image but a set of abstract constraints. That is where gradient-free methods might actually win.

---

Source: Fitting Neural Textures and PBR Material Maps with ES (No Backprop)