Hi Community. I have query regarding cloud offering. As I see that first 20000 tasks are free and later are charged based on usages. But I am not able to find other details like system limits (memory, cpu, network, storage and runtime limits)

Because there are no limits 😄 Prefect has a hybrid execution model which keeps your code and data private running entirely on your infrastructure. Therefore, Prefect hosts all the orchestration components for you (the database, API, auth, RBAC, scheduler, UI, and other services required for the orchestration API) but the actual execution layer for your flows is entirely within the customer's infrastructure. And if you are using some cloud provider, then the only limit is how much you want to spend on the cloud provider's infrastructure 🙂 Check out this page that tells more https://www.prefect.io/why-prefect/hybrid-model/

Thanks Anna. I will check it out. I was not able to find it on main website.

also just to mention: we also have a Discourse forum and there is this topic that provides various materials that may help you get started with Prefect https://discourse.prefect.io/t/getting-started-with-prefect/27

Nice information and business model. It was insightful. Just curious about plan to support serverless (similar to aws step functions)?

Check out this blog post hot off the press 😄 https://www.prefect.io/blog/easy-event-driven-workflows-ith-aws-lambda-v2-featuring-serverless-framework/

awesome 🙏

@Anna Geller thanks for the post. I have multiple workflows that follow this pattern: • do some tasks • trigger an external workflow • wait for the external workflow to complete, and pass me the output • do stuff with the output In my opinion it makes sense to be able to express this entire workflow as one prefect flow - vs. break it into two flows (up to the external point and after the external point). So an I right in concluding that the recommended way to do this is to do a task that “sleeps” and polls the state of the external condition? I guess this is the “sweet spot” of cadence/temporal style tools…

The workflow you described falls more into the "AWS waiter" category when you submit something like AWS Batch job and poll the API for the job completion status - polling always means making some API call, then the task "sleeps" before making another call and the cycle repeats until the condition is met, i.e. the return job status code indicates that this external job completed. You're right that this pattern is more like "sensors" in other workflow tools due to this polling mechanism. I updated this topic's section based on your answer https://discourse.prefect.io/t/is-there-an-equivalent-to-sensors-in-prefect-how-do-i-trigger-event-driven-workflows/76#h-2-leveragin[…]bstractions-2 But all those patterns are totally valid and supported by Prefect, depending on your use case. LMK if you have any other questions about that.

@Anna Geller thanks. Since • I have to wait for days/weeks for the external signal to happen • the external signal is quite specific I was thinking of something like this: 1. do some tasks 2. call external workflow that takes weeks 3. task that processes the external workflow’s result: • if there is no file on S3 (outcome of the weeks-long work from (2) - put itself in PAUSE state • if the file is there, process it 4. Some external event (e.g. lambda, webhook) happens when the workflow from (2) is completed 5. The external event from (4) causes task (3) to RESUME. My main question is - what do I need to do in order to implement 5? Just call prefect API and mutate the state of the task from PAUSE to RESUME? or perhaps follow something like this blog post? Or, Is there perhaps a more elegant way to raise an event from outside of prefect that will cause the workflow to resume?

It if takes weeks, it's better to just call a subflow via API call once this external job finishes rather than having some idle/paused workflows waiting and polling for status for weeks - it makes things much easier to troubleshoot and observe. What is your flow doing that takes weeks to process if I may ask?

labeling images by humans

Can you explain your use case more? Maybe there's a better way of approaching it

1. Query DB 2. Call external labeling platform (e.g. labelbox) API and create a human image labeling assignment 3. wait for the humans to label the images 4. Get a “hook” call from the labeling platform 5. Process the labels and upload to DB

thanks for sharing your use case! I agree, the way you explained it, it seems that a manual approval pattern is what you need. I was a little worried when you said your workflow takes weeks to execute - I thought this was some terribly inefficient sequential compute, but the human intervention here that requires a pause in between explains a lot and clears up my confusion 🙂 Check out those resources and let us know if you have any questions: • https://docs.prefect.io/core/idioms/pause-for-approval.html • https://medium.com/the-prefect-blog/needs-approval-184f2512a3cf

@Anna Geller thanks, and if I want the “manual approval” to come from a webhook, do I just mutate the state of the task to “resume” via graphql?

You certainly could do it via GraphQL API mutation. How do you trigger the approval/resume step? Do you have some custom extra API and UI when someone clicks a button to approve and resume the workflow? The way it was designed to be used was that you don't need to build any custom webhook or GUI and you can just approve directly from the UI

@Anna Geller yes something generates the webhook condition that tells me i can continue. I would like to avoid another step of manual approval to continue - just to continue as soon as this event fires outside of prefect. Also I think that for the general case, even manual approval shouldn’t necessarily require logging into prefect, since the approval can be a CI step (e.g. QA) or other non-data eng persona. Anyhow I will give it a try. Much appreciated!