Sven Teresniak
07/27/2020, 6:41 PM$PREFECT__HOME_DIR
. I see my pickled flows and a lot of pickled results. But Prefect seems not to be aware of that.
The last log line is btw Flow 'session-creation-watchdog': Handling state change from Scheduled to Running
. In a successful run, the next log line would be the start of my first task.
I don't know where to start to debug that. What can I do to better understand the problem? Is there a timeout or can I force a "PING" to the flow to identify the real state?Kyle Moon-Wright
07/27/2020, 7:07 PMSven Teresniak
07/27/2020, 7:27 PMKyle Moon-Wright
07/27/2020, 7:30 PMSven Teresniak
07/27/2020, 7:30 PMMarwan Sarieddine
08/15/2020, 6:38 PMSven Teresniak
08/15/2020, 6:58 PMn
time units using the --lifetime
CLI argument (see https://docs.dask.org/en/latest/setup/cli.html#cmdoption-dask-worker-lifetime ). But that's not good because it will cause running tasks to fail.
When you put your boilerplate code (for flows) into separate modules, its hard to deploy new boilerplate because its not trivial in Python to "un-import" && re-import (boilerplate) packages. So I need an easy and robust way to (re-)start Dask Worker (or to command the nanny process to spawn worker).
To some degree the behaviour of (standalone) Apache Spark should be adapted to Prefect-Dask: every flow spawns its worker (executor in spark), use them for tasks, exits them after some idle time (or explicitly somehow).
I know, its much easier to spawn, manage and isolate tasks and flows using Docker or K8s. But in my company its not an option for a container to modify the state of its own container management (k8s, docker).
Maybe I'm "holding it wrong" but when Dask workers are stalled, more and more flows are in status RUNNING (we want to schedule a LOT of work with Prefect). This is a clearly a negative data engineering pattern. 😞Marwan Sarieddine
08/15/2020, 7:07 PMJeff Brainerd
08/15/2020, 11:18 PM