Is there any way to see in the cloud ui what executor the flow is registered with? I'm not seeing th...

Trevor Kramer

05/04/2021, 1:27 AM

Is there any way to see in the cloud ui what executor the flow is registered with? I'm not seeing the task concurrency I am expecting and want to confirm the executor settings.

Kevin Kho

05/04/2021, 2:48 AM

Hi @Trevor Kramer, I'll double check tomorrow but it doesn't seem there is a way unfortunately. What issues with task concurrency are you seeing?

Trevor Kramer

05/04/2021, 3:18 PM

I have a mapped task with 20 segments and they are executing sequentially. I am configuring the executor this way.

Copy code

flow.executor = DaskExecutor(cluster_kwargs={
    'n_workers': 10
})
flow.register(project_name=project, labels=labels, idempotency_key=idempotency_key, no_url=True)

Kevin Kho

05/04/2021, 3:21 PM

Is it 1 mapped task with 20 different inputs? Or you do mean a mapped task that goes 20 tasks deep?

Kevin Kho

05/04/2021, 3:23 PM

And you have no parallelization at all? do you see the same behavior with LocalDaskExecutor?

Trevor Kramer

05/04/2021, 3:36 PM

1 mapped task with 20 inputs

Trevor Kramer

05/04/2021, 3:36 PM

i'm testing the LocalDaskExecutor now

Kevin Kho

05/04/2021, 3:38 PM

Might need to remove the idempotency key for a bit

Trevor Kramer

05/04/2021, 3:38 PM

Is there a way to send you a file privately? I have a log file which shows the behavior

Kevin Kho

05/04/2021, 3:38 PM

Yep DM me.

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.