j

    Jai P

    3 months ago
    👋 hullo! i'm curious, if i want to use prefect 2.0 for workflows that i'd typically do with something like celery (e.g. enabling an event-driven architecture vs. more ETL-y type workloads), is that a bad idea?
    Kevin Kho

    Kevin Kho

    3 months ago
    No it’s not and it’s something we think 2.0 can handle. The only thing is that Prefect is not resource-aware so you need to explicitly set the flow concurrency yourself for you agent. Does that make sense? You can use the API to create a deployment
    j

    Jai P

    3 months ago
    hmm, i think that makes sense! and i'd set the flow concurrency in the deployment spec?
    oh, or is it
    work queue concurrency
    that i'd have to set
    Kevin Kho

    Kevin Kho

    3 months ago
    Yes exactly
    j

    Jai P

    3 months ago
    awesome! is there anything else you'd suggest we look out for? to paint a picture, we have an entity that might trigger 100s of events per minute that we'd like to trigger flow runs for. generally we'd just put that behind a queue and have a celery app process the different types of events, but we have a need for things like caching/more fine-grained concurrency, and don't necessarily want to implement those features ourselves (and think using prefect could be a better solution)
    Kevin Kho

    Kevin Kho

    3 months ago
    I think nothing else aside from being aware that Prefect might error if it tries to deploy a flow but doesn’t have resources
    j

    Jai P

    3 months ago
    got it, ok! if we run into problems along the way, we'll definitely reach out. thanks for the help!