Unless something has gone very sideways, there should be only one not-quite-VM involved in running Linux containers using Docker Desktop for Windows; the one which runs WSL. Docker runs every container as a Linux namespace (like regular docker) inside a specialized WSL 2 distro, each of which also runs as a Linux namespace inside a single Linux root.
This memory it uses is mostly swappable. You can check this with Process Explorer or the like - look at the "vmmemWSL" process when it's idle and you can see the working set drop back.
It usually doesn't shrink much below 2GB because it doesn't free the Linux page cache until the Linux kernel frees it, but you can poke this manually with `echo 1 > /proc/sys/vm/drop_caches`
Here for more details: https://devblogs.microsoft.com/commandline/memory-reclaim-in-the-windows-subsystem-for-linux-2/
We’ve added a new Windows Subsystem for Linux (WSL) feature in Windows Insider Preview Build 19013: Memory Reclamation for WSL 2! Previously, when the memory needs of the WSL2 Virtual Machine (VM) would grow, either from your workflow or by the Linux kernel,