NikoMar 23

hi there rust peeps

does anyone have any clue why a tokio runtime would have a hard cap on the number of tasks at 32,768?

hitting this limit silently fails to spawn new tasks and existing tasks hit "device or resource busy" errors when they need to use IO

no ulimits have been hit as far as i can tell (83 open files, well under limit of 16,384, and all other limits are either unlimited or unrelated to this issue)

given the failure mode is completely silent i have no clue where to start debugging this

Show threadprincess pancake ​Mar 23
@niko Have you tried turning off io_uring if you use that?
Show threadNikoMar 23
@natty i do not use that
Show threadprincess pancake ​Mar 23
@niko are you perhaps watching filesystem changes or something? Is it Linux?
Show threadNikoMar 23
@natty this is linux, this is basically exclusively network IO, little to nil disk usage
Show threadprincess pancake ​Mar 23
@niko What do the spawned tasks do?
Show threadNikoMar 23
@natty mostly ship off data to other services and then wait for that to be processed
blocking CPU bound stuff is done on sync threads (spawned separately from the runtime) with queue communication between the two (it’s audio decoding), all of this is in a separate library and i don’t know of any user at larger scales than i with any issue like this
Show threadprincess pancake ​Mar 23
@niko silly question but did you perhaps run out of ports ​​
Show threadARGVMI~1.PIF

@natty

There are 32k ephemeral port numbers, so… 🤔

Edit: Wait no, the default on Linux is 28232 ephemeral ports, i.e. [32768, 60999]. Also, there are probably at least a few other processes using at least a few ephemeral ports at any given time, so you wouldn't get all of them to yourself.

@niko

Mar 23 at 3:00pmWeb