Hunter HillegasIn my first try using the Foundation Model framework, getting rate limited like crazy.
Mitch Winn@hunter Wait the local model is rate limited?
@mitchwinn It’s possible I’m holding it wrong since I’m just getting started but it’s throwing “rate limit exceeded” errors for me when I try to summarize a bunch of items in a loop, yes.
@hunter Interesting, I assumed one of the benefits of a local model would be that you wouldn't have limits like that. I know there is a context limit, maybe that is what you are hitting but the error message is just worded weird.
@mitchwinn Looks like the model itself technically doesn’t have a rate limit but the content guardrail service they put in front does. But that’s effectively a rate limit on the model so…
@hunter Yeah I have hit those guardrail errors as well, but got around it if I tweaked my prompt to be more descriptive or just used different words. I wonder if that will improve through the betas to be more descriptive on what is actually happening when those errors are thrown.
@mitchwinn My use case is news summarization so I don’t control the content input. If they don’t tweak things a bit, I don’t think I can use it. Too bad!
@hunter Ah yeah thats a bummer especially since it is just for summarization. I wonder if/when they allow us to use private cloud compute or other services if those guardrails will lessen. It would be a good question to ask to see if there is a way around it with the current APIs if you are attending the AI frameworks group lab later today!
@hunter Are you giving your LanguageModelSession instructions that tell it to be neutral? I quickly tried this and was successful summarizing a random politics focused article from NYTimes.
Here are the instructions I fed it:
"Please summarize the main points and key insights of the following news article in a concise and neutral manner. Focus on the essentiall details and avoid including opinions or speculative content. Keep the summary to one paragraph or less."
@hunter here is a gist of the code as well https://gist.github.com/mitchwinn/8b2b0fe507c141b2e61e5e396c491030
@mitchwinn Yes, though not exactly the same instructions. I’ve tweaked them a bunch without success but I will try your version in the gist. Thanks!
@mitchwinn Interesting - my instructions weren’t all that different but I do get some results with yours that were rejected with mine so hopefully that means I can work around most issues with a bunch more instruction tweaking.
@hunter Nice! This is all a bit of a moving target trying to find the exact words to do what you want. One thing I've learned from others regarding prompting, is to use AI to generate prompts for you. It works incredibly well and is what I did in this situation. It tends to know how to speak to itself!
@mitchwinn Yes! Do that often. Most of my work is with bigger, smarter models so I may just need to be way more explicit with this little guy.