Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Has anyone experimented with rigging prefect flows which execute <https://docs.prefect.io/api/latest/tasks/jupyter.html#executenotebook|jupyter notebooks>? We have folks who author scrapes in a no-code tool, and then they may need to set up scheduled postprocessing of incremental ingested data. Anyone doing something similar?

Hi <@U02ACJP2GLR>, have you seen the jupyter <https://docs.prefect.io/api/latest/tasks/jupyter.html|tasks> ? these use papermill under the hood to execute jupyter notebooks.

yup, that’s what i think i linked to in the question… i was wondering if folks have set up flows that publish a notebook to s3 and then run it… (eg we use <http://deepnote.com|deepnote.com>

Oh sorry I missed that. I see what you mean. So not quite the same, but I did see a user run a flow on deepnote and he has a blog entry <https://prefect-community.slack.com/archives/CL09KU1K7/p1629196082026500?thread_ts=1629162913.011500&amp;cid=CL09KU1K7|here>

But I think I took a look around Deepnote and there didn’t seem to be an API to interact with? Prefect would need to orchestrate that through an API

<@U01QEJ9PP53> ask and one shall receive — the good folks at Deepnote are working on an api: <https://community.deepnote.com/c/beta-testers/notebooks-api-early-access>?

Nice! That would open a lot of doors for orchestration