Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
p
Preston Marshall
02/07/2020, 11:14 PMSo I'm using the DaskExecutor with a task that has a map() over a list of files. I would expect this to be done in parallel but it runs serially. How do I allow a flow to run in parallel?
c
Chris White
02/07/2020, 11:15 PMcan you share you code? Also, how do you know the tasks are still running serially?
p
Preston Marshall
02/07/2020, 11:18 PMI'm just watching the logs, and it goes through one by one
can I dm you?
dm-ed you the code
c
Chris White
02/07/2020, 11:20 PMso my guess is that
start_clusterp
Preston Marshall
02/07/2020, 11:21 PMI can share that, it's just the code from their docs wrapped in a function
from dask_kubernetes import KubeCluster
def start_cluster():
cluster = KubeCluster.from_yaml('worker-spec.yml')
cluster.scale_up(10) # specify number of nodes explicitly
cluster.adapt(minimum=1, maximum=100)
return clusterwhen it starts up I notice it makes 10 and then immediately tears them down
in k8s
c
Chris White
02/07/2020, 11:22 PMhm strange; can you try removing the k8s piece and just use the following executor:
executor = DaskExecutor()p
Preston Marshall
02/07/2020, 11:22 PMsure, just try it locally?
c
Chris White
02/07/2020, 11:22 PMyup yup
p
Preston Marshall
02/07/2020, 11:23 PMthat runs in parallel
c
Chris White
02/07/2020, 11:24 PMgotcha; so there’s some issue with the kubernetes dask cluster you’re spinning up
does your worker spec specify
nthreadsnprocsp
Preston Marshall
02/07/2020, 11:26 PMso, if I take away the scale_up() call, it starts out with 1 and then eventually spins up what looks like 100 pods
the other weird thing is that those initial pods from the scale_up immediately die
I'm not specifying anything you don't see
c
Chris White
02/07/2020, 11:27 PMInteresting - I recommend opening an issue on the dask-kubernetes repo with the behavior you’re seeing
p
Preston Marshall
02/07/2020, 11:27 PMoh wait, sorry. I am specifying nthreads
c
Chris White
02/07/2020, 11:28 PMare you fixing it to 1?
p
Preston Marshall
02/07/2020, 11:28 PM2
maybe it just looks serial, and is actually 2 at a time. the scale_up thing is odd though
c
Chris White
02/07/2020, 11:29 PMhm i’d expect to see 2 tasks running simultaneously in that case
p
Preston Marshall
02/07/2020, 11:29 PMyeah it may be, they are slow
c
Chris White
02/07/2020, 11:29 PMgotcha
yea the scale_up issue you should open a dask-kubernetes issue for
p
Preston Marshall
02/07/2020, 11:30 PMideally I'd have some prewarmed pods so yeah I may do that
thanks!
c
Chris White
02/07/2020, 11:30 PManytime!
p
Preston Marshall
02/08/2020, 12:55 AMit seems like because my minimum was 1, it was killing the extra pods because it didn't have any load and I had adapt(min=1)
c
Chris White
02/08/2020, 1:02 AMGotcha, that makes sense