VissFeb 26

my stupid llm research is absofuckinglutely not going the way i was hoping.

ive spent like a fucking week trying to setup a testing harness to get local models to do the same test 100 times, aperture science style, to test the drift of their results

but 100% of the time, the model:
- emits tool calls incorrectly, so i see them
- ignores instructions
- falls into a loop
- says its gonna do stuff, then .. just doesnt
- intentionally deviates from instructions even when explicitly told not to

Show threadPaul_IPv6

@Viss

sounds like it's still in the adolescent stage ;)

maybe threaten to take away its internet privileges for a week?

Feb 27 at 12:01amWeb
Show threadschrotthaufenFeb 27
@paul_ipv6 @Viss You joke, but the other day I spoke with a colleague who did some research on LLM reliability before deciding doing a PhD wasn’t for them. They said threatening the model did slightly improve results across the board.
Show threadPaul_IPv6Feb 27

@schrotthaufen @Viss

i was only half joking.

LLMs resort to use of threats and blackmail, so not shocking they respond to them too.

personally, i think "human-like" is *NOT* a feature in software...