I'm almost there but now I have a mismatch between the sprite projection and the cylinder.

The cylinder uses raycasts (it basically interpolates between the leftmost and rightmost rays of a theorical frustum) but the sprite uses a world->clip matrix which should be that frustum but I don't get the same results for now.