Hello all, nice to meet you! Is there a channel for the new

ControlFlow

AI workflow framework (https://controlflow.ai/welcome), or where do I go with questions? (specifically I can’t seem to find information on observability, so it is not apparent how to trace the data that are sent to/received from the LLM’s, how many tokens that are used, or how to keep track of aggregate costs for the model inference in the flows.

Hey Poe! We should probably make a channel for this, haha. We're still building this out, and your feedback at this point can dramatically shape where we prioritize the "LLMOps" observability (as distinct from the "agentic" part). Happy to discuss here but would love to invite others on Github to join the discussion so we can all build this right. Mind opening an issue on Github? Happy to have this conversation wherever but that has the most eyes on it for now.

Hi Adam, thanks - Sure I’ll open an issue on github, I just wanted to make sure I didn’t miss anything in the docs before I made one.

Nah nah you were absolutely right to ask here, and I appreciate you doing that

what you mean LLM observability or issue tracker?

LLM"Ops" Observability like tokens / cost, etc. Woof hope we never implement our own issue tracker.

I thought so - have a look at MS AutoGen, they made it fairly easy

🫡 sweet, thanks for the pointer! What do you like about how they did it, what do you not love about it?

You can just access all the data directly in the response object and keep track of everything, very convenient.

🙇 . We're langchain deeply under the hood I'm sure there are some low level hooks like that we can bring a bit more to the surface for sure

Yes surely, I know these are early days but this question on Ops will a popular one, also understanding how you combine all the prompting with the context and scope to bigger prompts is going to be something that people want to understand and have more control over, I’m sure. Btw I love the whole controlflow premise with data types and more atomic process steps, it is definitely the way forward

Yeah, 💯. Right now it's a lot of primitives welded together, but genuinely @Jeremiah and I's next focus is to bring total transparency to how everything gets assembled and give folks the chance to customize it at any level // take full control.

I think a big first step is to provide optional access to the full task prompt history. That would be really good.

👌👌 @Poe thanks for the feedback here. We're currently working through a new backend for tracking a lot of things -- not just the usage metadata but even our internal actions so we can do some really cool things with how we show agents their universe. I think storing all of these details as message attributes (as we do today) is woefully insufficient and we can make it way more accessible. Not ready to say what the timeline will be but it is the thing I'm working on right now!

Haha, being able to load just messages related to a single task is a good example -- because the agents can work on multiple tasks at once, we don't have a natural place to track that! (But are starting to track it in the new pipeline). In the meantime though, try

flow.get_messages()

after you run it, I think it'll give you the whole thread history (not broken out by task though)

Ok sounds good, just keep in mind if you want quick onboarding that developers like to be able to debug directly in the IDE so you don’t have to dig into logs and db’s or api’s to find the data you are looking for 🙂

That would be making it too easy for you. 😂

😄

Haha totally agree

As we iterate on this @Poe are you down to keep giving us feedback until these edges are smoothly sanded?

Awesome ❤️

🙌

As we iterate on this @Poe are you down to keep giving us feedback until these edges are smoothly sanded?

I appreciate the offer. Lets talk later DM. I’d like to understand your vision.