Lesson learned:

When writing a wrapper future or a wrapper stream, always remember the waker contract. When a future registers a waker and that waker fires, the runtime polls the outermost future, which eventually polls the wrapper. If your future swallows the poll without forwarding it, you’ll run into the same class of problems

https://www.e6data.com/blog/deadlocking-tokio-mutex-without-holding-lock