Explosion 馃挜

We are really excited to share that we have just released the alpha version of Prodigy v1.12! This includes LLM-assisted workflows for data annotation and prompt engineering as well as extended, fully customizable support for multi-annotator workflows.

https://support.prodi.gy/t/prodigy-1-12-alpha-release-llm-assisted-workflows-prompt-engineering-fully-custom-task-routing-for-multi-annotator-scenarios/6552

Prodigy 1.12 alpha release: LLM-assisted workflows, prompt engineering & fully custom task routing for multi-annotator scenarios.

Hey everyone! We are really excited to share that we have just released the alpha version of Prodigy v1.12! (v1.12a1). This release is available for download for all v.1.11.x license holders and includes: New recipes for LLM-assisted annotations and prompt engineering: the LLM assisted workflows we have announced a while ago are now fully integrated with Prodigy and available out of the box. For v1.12a1 you'd still be restricted to OpenAI API to use them, but by v1.12a2 we definitely want to...

Prodigy Support
May 22, 2023 at 3:24pmWeb
Show threadExplosion 馃挜May 22, 2023

LLM-assisted workflows for bootstrapping ner and textcat labels are now available out of the box. For now, we only support GPT3, but by v1.12a2 we definitely want to leverage spaCy-llm to enable more flexibility, notably the use of open source models.

https://github.com/explosion/spacy-llm

GitHub - explosion/spacy-llm: 馃 Integrating LLMs into structured NLP pipelines

馃 Integrating LLMs into structured NLP pipelines. Contribute to explosion/spacy-llm development by creating an account on GitHub.

GitHub
Show threadExplosion 馃挜May 22, 2023

We present a brand new workflow for prompt engineering that allows you to compare the quality of several prompts in a tournament. The algorithm uses the Glico ranking system [https://en.wikipedia.org/wiki/Glicko_rating_system] to select the best prompt.

https://future--prodi-gy.netlify.app/docs/large-language-models#tournaments

Glicko rating system - Wikipedia

Show threadExplosion 馃挜May 22, 2023

In v1.12 onward Prodigy will expose a task_router component that accepts a custom Python function that determines how tasks should be distributed across annotators. You could, for example, route tasks based on model confidence.

https://future--prodi-gy.netlify.app/docs/task-routing#task-routing

Task Routing 路 Prodigy 路 An annotation tool for AI, Machine Learning & NLP

A downloadable annotation tool for NLP and computer vision tasks such as named entity recognition, text classification, object detection, image segmentation, A/B evaluation and more.

Prodigy
Show threadExplosion 馃挜May 22, 2023
Apart from custom routers, you can also use one of the built-in ones, which you can configure from the config file. Specifically, you can now specify partial overlap by determining the average number of expected annotations per task by putting the following in your prodigy.json
Show threadExplosion 馃挜May 22, 2023

Similarly, you can now customize the way each session is initialized by providing a custom session_factory callback to the Controller. You could, for example, count progress based on session attributes.

https://future--prodi-gy.netlify.app/docs/custom-recipes#session_factory

Custom Recipes 路 Prodigy 路 An annotation tool for AI, Machine Learning & NLP

A downloadable annotation tool for NLP and computer vision tasks such as named entity recognition, text classification, object detection, image segmentation, A/B evaluation and more.

Prodigy
Show threadExplosion 馃挜May 22, 2023

We're super excited to release these features to get feedback but there are a bunch more features underway. We're hoping to release v1.12a2 soon, which will add more features to this list!

Prodigy v1.12a1 is available to all v1.11 license holders.

To install run:
pip install prodigy==1.12a1 -f https://[email protected]

Supports macOS, Linux and Windows and can be installed on Python 3.8 and above