https://prefect.io
Join Slack
Anyone know if the `LocalAgent` can run using the `LocalDaskExecutor`? It’s parallelizing stuff fine...
# ask-community
j
Anyone know if the 
LocalAgent
can run using the 
LocalDaskExecutor
? It’s parallelizing stuff fine locally when I just run 
flow.run
but on a server running a local agent it looks like it’s going one task at a time (I’m just watching the logs in prefect cloud). I’m setting 
flow.executor
before registering it, if that matters. I don’t see any indication in Prefect Cloud of what the flow’s executor is.
👀 1
k
Yes it can. May be try setting 
n_workers
explicitly to make sure it uses the available cores.
Oops I mean 
num_workers
.
Copy code
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
taken from here
j
Right I’m setting it to 4, lemme dig deeper to see if I notice anything
k
Oh I think I know what might be happening. Do you define your executor in a different file than your Flow?
j
I do yeah
It’s like a generic “register all flows” CLI that we have
k
The executor is loaded from the storage. It’s not serialized along with the Flow because it can contain private stuff like the Dask address so we don’t keep it.
Executor specifically needs to be in the Flow file
j
🙌 got it. thank you @Kevin Kho
👍 1
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by