Hey Prefect community 😎 had a question about local flow execution that I'm hoping someone can help with. Before the local prefect runner, we were previously running/debugging flows locally like this:
Copy code
import os
from flows.wherever import flow
from prefect.executors import LocalDaskExecutor
from prefect.storage import Local
context = {
# ...
}
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
flow.storage = Local()
state = flow.run(
param="something"
context=context,
)
This works fine, but doesn't seem to fly with flows that call
StartFlowRun
, since it's trying to reach out to a server. Anyone have a way to do dependent flows locally? Not sure if the new local runner can do it, I haven't had the chance to try it yet 🙂
k
Kevin Kho
09/29/2021, 8:37 PM
To actually run it, some other users spin up Prefect Server for local dev purposes
j
Jason Bertman
09/30/2021, 12:13 PM
Gotcha, so atm there's no easy way to do it locally without spinning up a local server?
k
Kevin Kho
09/30/2021, 1:52 PM
Unfortunately not because that
StartFlowRun
code is calling the API using the Client under the hood. For unit testing, it would need to be mocked. Maybe you could mock the behavior to import the flow locally then use the
flow.run()
?
j
Jason Bertman
09/30/2021, 4:35 PM
Had the same thought - I'll go down that rabbit hole for a bit, thanks!
Jason Bertman
09/30/2021, 7:41 PM
For any others who are wondering, this methodology works fine. Wrote a custom `StartFlowRun`task that detects some of our local contexts, finds the relevant flow, and
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.