I am excited to play around with Controlflow but have a question based on my (very brief) quick scan of the docs. Over the summer, I had a project where I built out a flow that allowed users to upload files and ask questions, compare, whatever. For this, I used the Assistants API from OpenAI to handle threads, files, whatever. The project was setup to be very modular, in that conversation threads were created, conditionally files were handled, and then assistants invoked. It was clumsy but worked. I want to use that project to frame out my learning in Controlflow, but it's not obvious to me how it would to wire that in given some "tasks" are for API calls and not actually invoking an LLM. I am sure I am missing something obvious, but any references to docs or nudges would be greatly appreicated.

@Marvin: i'm developing in a git branch and i've made changes affecting various flows. I would like to test that these flows run correctly when deployed and run on cloud infrastructure before actually merging my branch into main. Right now, this process is very manual: for each flow I have to create a new deployment in my deployment.yml that points to my current branch. Worse still, I have to account for that everywhere I call

run_deployment

in my code. Can you provide guidance on how to make this type of testing more automatic?

@Marvin can you show me an example of a prefect task that uses subprocess to call a command line program in a way that doesn't conflict with prefect

Hi I'm trying to use a bitbucket storage block with a private repo. I've setup a bitbucket credentials block which is authenticated like this, using username and password. Then I use that inside a bitbucket repository block. But I get this error:

token = self.bitbucket_credentials.token.get_secret_value() AttributeError: 'NoneType' object has no attribute 'get_secret_value'

does anyone have any ideas? The blocks UI does suggest personal access token is optional. Unless it's 'optional' in case it's a public repo

Here’s a view of the space needle on new years!

Is there a way to start a prefect agent via code instead of manually using the CLI?

Hi everyone. I'd like to understand how memory is managed in a flow. I have a long-running flow that calls an API to get data. The flow works roughly like this: • get list of URLs to retrieve (about 180k URLs) • break the URLs in groups of 150 (list of lists) • a mapped task receives a list of 150 URLs and calls the API • another mapped task receives the API output for 150 URLs and saves the output to s3 I am using S3 results caching for the data intensive tasks (tasks 2 and 3 above) and Prefect Results for the rest of the tasks. I am seeing that the memory utilization keeps increasing until the container runs out of RAM (this is running on ECS fargate). It seems to be keeping the data retrieved from the API in memory, even after it's saved to s3. I can increase the container RAM but am trying to understand how I could write the flow so that it does not run out of RAM. This is what the Memory Utilization chart looks like. Eventually, the container dies and Prefect Cloud restarts it. Any suggestions?

Hi, is there a way to iteratively run a flow for batch processing over a large dataset? Here’s the flow I’m trying to achieve: 1. Start flow with 3 parameters: start timestamp (ts_start), batch size (n), end timestamp (ts_end) 2. Task extracts batch of n documents starting from ts_start 3. Cache nth document’s timestamp (ts_last) 4. Go back to (1) passing in ts_last as ts_start if ts_last < ts_end In other words, I want to iteratively process batches until we get a document with timestamp ts_end

Hi there, I'm new to Prefect and checking out options to see if Prefect will work for my company. I'm having trouble starting an ECS Agent within our infrastructure that communicates with PrefectCloud. We are using HIPAA compliant AWS architecture as described here, i.e. most of the infrastructure is in private VPCs and exposed via a public VPC. Can you help me figure out if that's my issue?

@Marvin I want to run sub-flows on separate pods to run in parallel, can you give me an examples ?