bruno.corucho
07/17/2020, 12:26 PM@task
def hello_world():
time.sleep(15)
logger = prefect.context.get("logger")
logger.info("I WOKE UP, GUYS!!!")
with Flow("strdata-test") as test:
hello_world()
hello_world()
hello_world()
hello_world()
hello_world()
for n in range(4):
test.storage = Docker(registry_url=os.environ["REGISTRY_URL"],
dockerfile="deploy/Dockerfile",
image_name="strdata-flow"
)
test.environment = DaskKubernetesEnvironment(max_workers=4)
test.register(project_name=os.environ["PROJECT_NAME"])
Output:josh
07/17/2020, 12:27 PMprefect_version='master'
on your Docker storage does it fix the issue?bruno.corucho
07/17/2020, 12:28 PMjosh
07/17/2020, 12:29 PMprefecthq/prefect:latest
corresponds to the most recent pointed release and prefecthq/prefect:master
corresponds to masterRobin
07/17/2020, 1:23 PMbruno.corucho
07/17/2020, 1:24 PMjosh
07/17/2020, 1:25 PMRobin
07/17/2020, 1:26 PMjosh
07/17/2020, 1:28 PMmax_workers=4
then it will start with 1 worker and dask will dynamically scale up to those 4 as it sees fit. If you want more workers at the start then do something like min_workers=4
bruno.corucho
07/17/2020, 1:28 PMjosh
07/17/2020, 1:33 PMr
tasks can be executed in parallel once the upstream t
tasks finishbruno.corucho
07/17/2020, 1:49 PMRobin
07/17/2020, 5:45 PM@task
def say_hello():
print("Hello, world! Got a secret for you :)")
sleep(10)
with Flow("simple-dask-kube-flow") as flow:
for i in range(30):
say_hello()
flow.environment = DaskKubernetesEnvironment(
min_workers=10, max_workers=100, labels=["k8s"]
)
flow.storage = Docker(registry_url="secreturl", prefect_version="master")
flow.register(project_name="eks_test_01")
So we would expect the dask cluster to spin up many workers and execute the tasks in parallel.
And indeed, dask creates many workers (see attached).
However, they are somehow cancelled and don’t run in parallel as can be seen on the gantt chart (see attached).
Weirdly, in the beginning there were even 4 parallel runs, but afterwards only seriel runs…
Any thoughts what could be the reasons?josh
07/17/2020, 5:58 PMRobin
07/17/2020, 6:04 PMnicholas
07/17/2020, 6:17 PMRobin
07/17/2020, 6:17 PMnicholas
07/17/2020, 6:18 PMMarvin
07/17/2020, 6:18 PMnicholas
07/17/2020, 6:18 PMRobin
07/17/2020, 6:24 PMnicholas
07/17/2020, 6:24 PMRobin
07/17/2020, 6:29 PMprefect 0.12.3+44.g0b3ef62bc.dirty
.
Might this cause the problem?
Unfortunately, my colleague teard down the EKS cluster, we might tear it up again later, or tomorrow (it’s 20:30 over here in Germany)josh
07/17/2020, 6:30 PM