@mjambon yeah I'm sure for my maps I make mistakes. All I do for error checking is to perform the forwards warping onto satellite imagery in QGIS, then visually check for discrepancies between the two. With this I can catch gross errors. E.g. overlay:
https://www.facebook.com/photo/?fbid=315155968310211

Beyond that, perhaps you could, on a per point basis, leave the point out of the referencing process, then see where it gets mapped to. If it is mapped far away from its reference location, perhaps the reference is wrong.

@mjambon I have a similar project at https://onamap.me/

For the lat/lon to pixels algorithm, I use a triangulation of the reference points to interpolate u,v pixel coords based on pixel coords of nearby reference points
https://turfjs.org/docs/api/tin

@verbeeld hi, very cool!

Wanted to share I've been doing something very similar since the start of last year.

I started doing this reverse geo referencing for hand-drawn tourist maps of towns, but also found it worked well for historical maps.

If anyone is in the UK, I've set up maps for London in 1561, Cambridge in 1575, and Oxford in 1605:

https://onamap.me/maps/London1561/
https://onamap.me/maps/Cambridge1575/
https://onamap.me/maps/Oxford1605/

I've been posting project updates on a facbook page:
https://www.facebook.com/me.onamap