Aider or llm integration?

[kuatroka]

Hi, can I use openlit to monitor and evaluate my prompt when using the "aider" AI Coding tool or Simon Willison's "llm" library? Thx

[patcher9]

Hey @kuatroka , Can you give some more context on what your are looking for?

[kuatroka]

Sure. I haven't used Openlit yet, but I was looking for a prompt management tool & evaluation and came across across it.

My use case is the following: I'd like to have a prompt tool that I can use integrated together with the main AI coding tools. For example, if I use "aider" AI Coding terminal tool or Cursor or Cline VS Code plugin, Codeium or "llm", the terminal AI access library, and when do my prompting in one of these tools, it would be good if they were tracked by Openlit, so I can go there and maybe add some checks or prompt rating or notes.For example if I used 4 prompts to get what I wanted, I'd mark 3 prompts as "fail" and the last one as "pass", so I grow my prompt library with the marks which prompts are good and bad.
The main idea is to have Openlit integrated behind the scene with the AI Coding tools (at least the currently most used) so I don't need to create my prompts in, let's say, "aider" and then copy/pate it to Openlit.
Thx

[patcher9]

To sum up what your ask is

If you make an LLM call using any of these coding tools, You want the prompt to be automatically collected and stored in OpenLIT (Only the ones you feel should be added or pass some checks, not all prompts)

as of now this wont be possible as we track LLM events when used via an SDK or in code and since these are tools, It wont be possible. I will reach out to some of these tool maintainers and ask to integrate OpenLIT into these tools. For now yeah you would need to copy/paste prompts into OpenLIT prompt hub.

lemme know if I didnt understand something yet

[abhijeetkaze]

Hi @patcher9, sharing my thought, let me know if its correct.

This use case sounds like @kuatroka needs a proxy solution. Since most tools/plugins/CLI apps likely won't have built-in instrumentation for OpenLIT or any other llm observability, you can leverage LiteLLM, which acts as a proxy for your llm. You can configure LiteLLM to send OpenTelemetry (OTel) logs to OpenLIT and then review them in OpenLIT UI.

And OpenLIT has manual tracing feature can then be used to store all traces along with prompts in the UI. This way, he won't need to copy/paste prompts manually. He can implement this by hosting your LiteLLM app with a custom configuration.

Here's an example setup:
LiteLLM Proxy Logging Documentation

So irrespective which plugin you use it will first go through lite llm proxy (if its supports litellm endpoints).

Issue might be that some plugins or apps we would need to check with them if they use litellm, In case of 'Aider' I checked it does have support for litellm Aider lite llm support

Additionally, LiteLLM already has integrations with other observability tools, It might also be worth exploring a contribution to LiteLLM for OpenLIT support, enabling deeper integration and making prompt management even more seamless.
Thanks

[patcher9]

@abhijeetkaze That sounds great, I agree this can be a way out for this situation. It should work! 🙌

Additionally, LiteLLM already has integrations with other observability tools, It might also be worth exploring a contribution to LiteLLM for OpenLIT support, enabling deeper integration and making prompt management even more seamless.

They do but only the Cloud based versions which have a dedicated endpoint, Since OpenLIT uses OpenTelemetry collector and can only be self-hosted, Right now the solution is to use OpenTelemetry integration in LiteLLM and as you said then route the data to OpenLIT

[kuatroka]

Sounds about right. Aider indeed works through LiteLLM and also it generates a hidden aider md file with all conversations: prompt and responses.
It's possible to switch models on the fly too and I don't know how this can be cougar by Openlit. I'm not too technical.
Thx