Stable diffusion seems to like very discriptive prompts, so I spelled out in more detail what I wanted to have: "a photo of a cute lil cat with a cute lil farmer's hat" (I thought it was important to write "lil") and I think now
#stablediffusion finally understand what I want. The cats on the left seem somwhat off, but I think the ones on the right are pretty spot on!