New blog post: making a transformer by hand, without training! Want to understand transformers and attention better? This post goes through assigning each weight for a GPT-2-like transformer to understand how they work.

Goes thru designing a simple tokenization scheme, embeddings, the qkv weights and attention head, and projecting that back to get 100% accuracy at predicting a simple sequence! It even has explainers for matrices and softmax if you're a little rusty :-)

https://vgel.me/posts/handmade-transformer

float z=-9.;vec2 F=FC.xy/r.xy-.5;for(int i=0;i++<30;){float X=F.x*cos(t)-z*sin(t),Z=F.x*sin(t)+z*cos(t),h=sqrt(X*X+F.y*F.y)-.2,d=sqrt(h*h+Z*Z)-.05;if(d<.01){vec3 n=normalize(vec3(X*h,F.y*h,Z*(h+.2)));o=vec4(.95,.5,mix(.8,.3,sign(n.z+.3)),1.)*vec4(.3+max(0.,n.y));break;}z+=d;}

// #sdf #rendering #glsl #donut

my singlepass c compiler (for a WIP blog post) is at 462 lines and I only have room for one of these, which would be more interesting?

(it already supports ints, pointers, arrays, if / while / do / for, functions, typedefs with the ~lexer hack~, and a few other things! goal is < 500 lines)

#blog #compiler

proposal for c 2.0:

int main() {
1 = 2;
if (1 == 2) return 1;
else if (*(&1) == 2) return 0;
else return 1;
}
// main() returns 0