@quixoticgeek I'm not saying that the concern is actually real, just that you're not specifically addressing it...
Personally the worst I've seen is a fatbike blast past close enough for me to say "jesus", but I've had that with an omafiets when I'm on foot too...
The only time I got into trouble, even during my first time on a bakfiets was because I got overloaded / panicked and didn't do the obviously correct thing (hesitated when i had right of way, forgot to have lights, or not just hitting the bell when somebody was in the path) - basically just a skill issue
The solution is just calming the fuck down and maintaining situational awareness and not being an asshole yourself.