Guess what, I went back to the drawing board and I got a perfect rendering* of the map now. The plus? Now I can change the viewpoint and it'll still works!
I also got some early culling done to skip cells from processing for rendering but I'll need to find less hacky ways if I want to play with the perspective.
* Until I find the next bug lol.