Mainly, when you’re messaging an iPhone user you get the benefits that it provides: less compressed images and videos, read receipts, more reliable messaging (you won’t see message “delivered” unless it actually reaches the person you’re messaging), encryption, etc.

When messaging someone without using iMessage such as when texting an Android user from iPhone or vice verca, it’s going directly over the data carrier networks, which heavily compresses images and videos, so if an Android user sends an iPhone user a video file directly, it ends up very blurry if it’s longer than 30 seconds-a minute or so, images are okay most of the time if they aren’t huge or detailed. You also don’t get read receipts on either end. The text also aren’t encrypted so the data provider and anyone who spoofs a cell tower can read all messages

iPhone to iPhone/iPad/Mac works great as you said though, but I actively use something else like Signal, Discord, or Telegram when messaging my friends who don’t have one because it’s a lesser experience on both ends.

The issue isn’t that iMessage is bad, it’s really good imo, it’s just that iMessage makes messaging people who don’t have iMessage worse by comparison, if it were opened up then everyone could text everyone without anyone missing out.