"solving cartpole... by evolving the raw bytes of a 1.4KB tflite microcontroller serialised model"
microcontroller models are so small you can just run a genetic algorithm directly against the bytes of the serialised model! :D
who needs gradient descent anways?
the main use case for dropout is as a form of regulariser. but what if we used it instead to a make a model that's robust to having different combos / forms of the input? we could then use a genetic algorithm to trade off input complexity to performance, all with a single model...
we usually batch training examples for a single model.
but what about instead passing a single instance through a batch of models?
with #jax this is trivial and means we can train ensemble nets directly.
self attention is a super important concept in a lot of neural architectures!
the description that really made it "click" for me was to think of it as a form of learnt map.
here's a 30min explainer starting with the first main problem solved with it and finishing with the "transformer" architecture
mat_kelcey