Hi @Chris White and team (and Prefect community), at the risk of opening a can of worms, I wanted to get any thoughts or recommendations you might have on developing Prefect flows in Jupyter notebooks. Our first Prefect use cases are focused on automating some of our data science workflows. We can certainly define task functions and flows in notebooks (following Prefect example code) but it feels like there might be an opportunity to do something more tailored to data science workflows. (Specifically, I'm thinking about whether it's possible to preserve some of the interactivity of notebooks while still defining tasks and flows, e.g. be able to run a single task independent of a flow, knowing that we may need to pass it appropriate parameters, etc.) It's definitely not required for us to do anything fancy, but since we're at the start of this journey I thought I'd at least ask before we start down any particular path.

This is a great question / discussion point; I definitely have some opinionated thoughts here but let me address a very specific need you mentioned:

Specifically, I’m thinking about whether it’s possible to preserve some of the interactivity of notebooks while still defining tasks and flows, e.g. be able to run a single task independent of a flow, knowing that we may need to pass it appropriate parameters, etc.

This is something that is very much possible right now. For example, you can run a Prefect Task in isolation by calling the

.run()

method of the task (and passing appropriate keyword arguments that it requires). You can also mock up Flow Runs by passing task states to

flow.run(task_states={...})

to test your triggers, etc. There are many other ways of running pieces of your Flow / Tasks interactively that I will make a point of documenting better in the very near future

@Chris White that makes sense on running Tasks / portions of a Flow interactively. Definitely will talk to you more as we get into it, but this gives us enough to go on for now. Thanks!