Hi...is there anybody here who could elaborate on the pros and cons of migrating from workflow engines popular in bioinformatics (e.g., Snakemake or Nextflow) to Prefect for large pipelines?

Hi! I can’t speak super directly to the differences relative to those engines myself, but just wanted to note that we have several bioinformatics users and that in particular @Kyle DeMerritt might be able to chime in at some point here from the side of having watched some of them migrate.

Thanks, that sounds like a good place to start then

I migrated a workflow from Nextflow, cons: Lack of reference flows like nf-core, bioinformaticians are familiar with Nextflow. There’s less magic going on and no DSL, pro or con depending on what you prefer. Might need to write custom code to limit concurrent tasks w.r.t system resources. Pros, easy unit testing (huge plus), the graph is defined and can be visualized before anything is run (quicker iteration). Tasks are much easier to reuse. And it’s Python which your organization is likely using already.

Thanks for your input. Sounds all good; so there is no strict limitation for Prefect that would prevent me from migrating a Snakemake workflow running on a cluster with PBS to Prefect (except for lacking reference resources maybe...)?

I haven’t used Snakemake but I expect that there isn’t any critical limitation. If dask can execute on your environment then you’re probably golden. One other pro I have to add is that Prefect looks and feels like it was made by seasoned veterans, a lot of the bioinformatics tools give off a “single committer” vibe

"a lot of the bioinformatics tools give off a “single committer” vibe" - exactly one of the reasons why I am looking for something more mature

Prefect will likely not be abandoned after someone gets their PhD 🙂

nailed it...