Just a thought, but I might be totally off - Do you have a spatial index?

I'm not familiar with SQLite, but I remember my friend having a similar issue when they had a lot of points in a postgres database (hour+ long waits for queries). After adding a spatial index, the spatial queries took seconds

https://postgis.net/workshops/postgis-intro/indexing.html