Mastodawn

RE: https://mathstodon.xyz/@11011110/116308526898756191

I think it is a bit funny (but only a bit) to observe the wild west that is happening at these big CS/ML conferences: Everything relies heavily on LLMs: the papers, the reviews, all paperwork, etc. Everybody is trying to trick everybody else. Dog eat dog.

Does anybody read the papers ever? Is this still science or just the game for an accepted submission at ICML which then secures a coveted job?

Show thread

Thomas Kahle 4d ago

BTW: The other day I tried to vibe-code exactly what they describe here: "We watermarked the PDF of each paper submitted with instructions, visible only to an LLM, instructing it to include the two selected phrases in the review. (A human reading the PDF would not directly see this watermark." While I could confirm that my tainted pdf did contain the watermark, using off-the-shelf Gemini would not act on my hidden instructions. The LLM seems to do pdf pre-processing and stick to visible text?

Show thread

Thomas Kahle

I was thinking to add this watermark to my own submissions, but then, I sometimes get referee reports where I wish the author had used a high-quality LLM to ensure at least a basic understanding of the paper before submitting their report.

I don't know what to make of all this. Also before LLMs peer review had problems with referees who are careless and abusive. These referees are empowered now, but the careful and thoughtful referees are also empowered, maybe?