Thread
#prefect-community
    p

    Peter Ebert

    2 years ago
    Hi...is there anybody here who could elaborate on the pros and cons of migrating from workflow engines popular in bioinformatics (e.g., Snakemake or Nextflow) to Prefect for large pipelines?
    Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    Hi! I can’t speak super directly to the differences relative to those engines myself, but just wanted to note that we have several bioinformatics users and that in particular @Kyle DeMerritt might be able to chime in at some point here from the side of having watched some of them migrate.
    p

    Peter Ebert

    2 years ago
    Thanks, that sounds like a good place to start then
    s

    StefanKjartansson

    2 years ago
    I migrated a workflow from Nextflow, cons: Lack of reference flows like nf-core, bioinformaticians are familiar with Nextflow. There’s less magic going on and no DSL, pro or con depending on what you prefer. Might need to write custom code to limit concurrent tasks w.r.t system resources. Pros, easy unit testing (huge plus), the graph is defined and can be visualized before anything is run (quicker iteration). Tasks are much easier to reuse. And it’s Python which your organization is likely using already.
    p

    Peter Ebert

    2 years ago
    Thanks for your input. Sounds all good; so there is no strict limitation for Prefect that would prevent me from migrating a Snakemake workflow running on a cluster with PBS to Prefect (except for lacking reference resources maybe...)?
    s

    StefanKjartansson

    2 years ago
    I haven’t used Snakemake but I expect that there isn’t any critical limitation. If dask can execute on your environment then you’re probably golden. One other pro I have to add is that Prefect looks and feels like it was made by seasoned veterans, a lot of the bioinformatics tools give off a “single committer” vibe
    p

    Peter Ebert

    2 years ago
    "a lot of the bioinformatics tools give off a “single committer” vibe" - exactly one of the reasons why I am looking for something more mature
    s

    StefanKjartansson

    2 years ago
    Prefect will likely not be abandoned after someone gets their PhD 🙂
    p

    Peter Ebert

    2 years ago
    nailed it...