Judith van StegerenToday I took a first look at LangSmith, a new platform for LLM production pipelines by LangChain.
I can't hook it up to a working pipeline yet because it's in closed beta, but it surely looks ambitious. It should make it easier to do logging, monitoring, debugging and evaluating pipelines (chains) against each other. It's tightly integrated with LangChain but it should support other frameworks/models as well.
The documentation is still a bit all over the place, but this walkthrough shows a sneak peek of what's possible:
https://github.com/langchain-ai/langchain/blob/master/docs/extras/guides/langsmith/walkthrough.ipynb
I especially like the idea of quickly evaluating variations of an LLM pipeline against a baseline of earlier runs.
However, I'm curious about what the volume of adoption will be. LangSmith is a platform, not a self-hosted opensource MLOps tool. Are engineers/users really willing to give all their data to yet another third party?
Some people already hesitate to use AI API's (such as the OpenAI GPT API) because they're concerned with leaking sensitive data -- let alone using third-party AI platforms that man-in-the-middle your LLM conversations.
#langsmith #llms #generativeai #mlops #ai #privacy #infoleaks
Maybe the fact that LangChain (the company) made it will lend LangSmith some legitimacy. LangChain (the framework) surely has seen a rapid rise in popularity -- although Github Stars might be a bad metric for actual production use.
Here is a comparison chart with some of the other popular NLP/LLM libraries -- these repos do not implement the same functionality but it should give you a rough idea.