@oli I get the right answer when I try. Same inputs.

@jesusmargar @oli and this is one of the problems with LLMs—they’re inherently stochastic

@jesusmargar @oli the inability to create reproducible test cases for these systems is an enormous problem for our ability to integrate them into other systems

@kevinriggle @oli surely they depend on a seed which depends on time?

@jesusmargar @oli somewhere. And sometimes it’s possible to start it from a known and fixed constant, and get the same results for the same prompt every time. (You can do this with some of the image generation models and invokeai iirc.) But in larger systems and longer interactions even with a fixed PRNG seed the path taken through the PRNG space matters, and small perturbations in it can create large changes in outcome

@jesusmargar @oli (ask unrelated questions A and B in that order, get good answers A’ and B’. Ask them in the order B and A, get the complete text of Atlas Shrugged)