Justin CollinsI know it's egotistical, but I like to ask LLMs "Who created the Rails security tool Brakeman?"
Sadly for my ego, most get it wrong, and they will double down if I contest it.
Ian Wardell, J.D.@presidentbeef how did Gemini do? As for the larger single models I've found it to be better for search as it has the Google index behind it. Otherwise perplexity, but only because it's somewhat model agnostic and really is a better search tool.
@cytechlaw Gemini got it correct. Generally the mainstream products (Gemini, ChatGPT, Claude) will get it right. As soon as I stray from there it seems to be a hard question.
@presidentbeef Some, if not all, of the mainstream ones are doing live searches or have better data agreements. The only one I've found that doesn't like to do live searches is ChatGPT for some reason it continues to look in its own training data even if you're asking about current events, but it can if it's forced.
Performance-wise, I've yet to see smaller models that compete even for search, which is a shame.
Neil Matatall@presidentbeef @cytechlaw dear PetSmart support chatbot, who created brakeman