Hey all! I wanted to share a really weird (but neat imo) Prefect workflow and task implementation that we tried just to potentially spark some ideas and discussions. Introducing ACTK (Automated Cell Toolkit) a pipeline to process field-of-view (FOV) microscopy images and generate features and render-ready products for the cells in each field. https://github.com/AllenCellModeling/actk Under the hood, each task in the workflow is managed by a custom Prefect Task object that we wrote that allows for git style management of large (multi-TB) imaging datasets while also managing the workflow run itself. This custom Prefect task object allows for each individual task in the whole workflow to: 1. run independently from the rest of the pipeline 2. manage produced data upload 3. manage produced data checkout 4. manage upstream data dependency download The underlying data storage and management implementation was written prior to Prefect adding

Result

and

Serializer

classes and so we are already thinking about how to upgrade those and have written quite a bit on potential implementation details so great job Prefect team for getting to it so quickly 🎉

This is super cool and interesting! I'm about to start an internship with people at the Carpenter lab at Broad Institute, the team who creates and maintain https://github.com/CellProfiler/CellProfiler. Do you guys already have a collaboration with them? Maybe we can work together!

Oh sweet! The Carpenter lab is awesome! We don't have any active collaborations with them but definitely cross circles and always have a great time. That should be a wonderful internship!

Very cool, thanks for sharing @Jackson Maxfield Brown!