Thomas Furmston
12/01/2021, 6:59 PMThomas Furmston
12/01/2021, 7:00 PMA -> B -> C -> D
. Now suppose I run the flow and tasks A
and B
complete, but C
fails because of a bug. I therefore want to fix the code in task C and redeploy the flow.Thomas Furmston
12/01/2021, 7:00 PMA
& B
are expensive to run, so I would rather not re-run them from scratch. Is it possible to re-run the flow with the newly deployed code from task C
onwards?Thomas Furmston
12/01/2021, 7:01 PMTilak Maddy
12/01/2021, 8:48 PMprefect auth login --key <MY_KEY>
even though my key was valid and authorized. I created so many keys later on still didn't work ....and in the end it turned out that I had already logged in with another key couple days ago. So i had to run prefect auth logout
before logging in with the new key .Tom Shaffner
12/01/2021, 8:55 PMprefect server config
to generate a docker compose file, and I'm partway down the path of trying to set that up to auto-start (still working through https://github.com/flavienbwk/prefect-docker-compose, which was linked in it), but so far it's not working and it seems like there should be a simpler answer here. Docker has restart policies already (https://docs.docker.com/config/containers/start-containers-automatically/), is there not some flag I can pass to prefect server start
to use them?Jason Motley
12/01/2021, 9:37 PMflow = Flow("myflow")
with flow:
df = extract(stuff)
print(df.head(10)) # This does not work
volkerjaenisch
12/01/2021, 10:54 PMhttps://upload.wikimedia.org/wikipedia/commons/8/89/Kozloduy_Nuclear_Power_Plant_-_Control_Room_of_Unit_5.jpg▾
brian
12/01/2021, 11:03 PMWilhelm Su
12/01/2021, 11:17 PMArun Giridharan
12/02/2021, 1:04 AM张强
12/02/2021, 6:20 AMBruno Murino
12/02/2021, 9:43 AMTilak Maddy
12/02/2021, 10:29 AMZheng Xie
12/02/2021, 10:52 AMprefect register --project yourproject -p yourflow.py
It seems that perfect creates a tmp{randomNumber}
folder and puts healthcheck.py and a Dockerfile, which participate the docker build.
But I accidentally deleted the tmp
folder, even after I recovered it, it still throws the following error. What happened was that Prefect expected a tmp
folder with a new random_number as the suffix. How can I recover from it?
https://stackoverflow.com/questions/70198040/error-in-prefect-when-build-docker-image-what-triggers-prefect-to-create-the-tmChris L.
12/02/2021, 11:01 AMClientError
when using Prefect Cloud for a mapped tasks. The inputs to each task are identical, but some mapped tasks are successful while a few raise DeprecationWarning: Using 'method_whitelist' with Retry is deprecated and will be removed in v2.0. Use 'allowed_methods' instead
(see screenshot for trace). Has anybody experienced something similar? I'm using urllib3 v1.26.7 and prefect
v0.15.9Aqib Fayyaz
12/02/2021, 11:33 AMhelm repo add prefecthq <https://prefecthq.github.io/server/>
i have cloned the repo
<https://github.com/PrefectHQ/server/tree/master/helm/prefect-server>
and so that i have this helm chart prefect-server locally and when i do helm install demo1 prefect-server i get the following error
Error: INSTALLATION FAILED: found in Chart.yaml, but missing in charts/ directory: postgresql
Tilak Maddy
12/02/2021, 11:53 AMflow.storage=Local()
to my Hello world flow ? Everything seems to be the samePinakpani Mukherjee
12/02/2021, 12:07 PMjcozar
12/02/2021, 1:01 PMArnon Kimhi
12/02/2021, 1:25 PMArnon Kimhi
12/02/2021, 1:27 PMif new_state.is_successful():
flow_run_id = prefect.context.flow_run_id
flow_run_view = FlowRunView.from_flow_run_id(flow_run_id)
b_task_run = flow_run_view.get_task_run("b-1")
Arnon Kimhi
12/02/2021, 1:28 PMJacob Blanco
12/02/2021, 1:30 PM{
task_run(
where: {_and: [{state: {_eq: "Running"}}, {flow_run: {state: {_neq: "Running"}}}]}
) {
id
start_time
}
}
Maybe unrelated but for tasks which have a lot of runs, the dashboard seems to hit some kind of query limit and crashes when you try to see the list of Task Runs. The stuck ghost tasks don’t always have a lot of task runs related to them.Tom Klein
12/02/2021, 2:20 PMPrefect
for the various benefits it could offer
i saw that there's support for "docker" and "kubernetes" tasks but since my DevOps knowledge is kind of limited i was wondering if (by any chance) there's some examples of that kind of usage laying around somewhere or if you could at least give your thoughts about whether what I'm thinking even makes sense?Sarita Patel
12/02/2021, 3:46 PMTom Shaffner
12/02/2021, 4:46 PMTom Klein
12/02/2021, 8:44 PMRun configuration
(Local, Universal, Docker, Kubernetes)
• an Agent
(e.g. Local, Kubernetes, Docker)
• and an Executor
- (Local, LocalDask, and DaskExecutor)
If i understand correctly -
• our Agent
is the entity (an always-running program, basically) which communicates with the Prefect server (and its scheduler, etc.) and brings jobs to execution phase. If it lives in Kubernetes then by default it executes flows as Kubernetes jobs , but - if we were to pick a "local" Run configuration
- it would run it on the agent itself?
• the Run Configuration
determines where the Agent
runs the flow (e.g., if we picked a Docker
run_configuration for our Docker
agent - then it would run as a docker image inside the docker image? or alongside it - as a "sibling" docker instance?)
• the Executor
determines how the flow should be executed, so for example if we had a LocalDask
executor running in a Docker
run_config with a Docker
Agent - it would spin up a local Dask cluster inside the docker that's running the job? but if we picked a DaskExecutor
then the flow would actually be executed outside the docker that's running the flow? (assuming our dask cluster runs alongside our Docker Daemon, e.g. on EC2)
did i get it right or am i missing something? 😄Tilak Maddy
12/02/2021, 10:19 PMBilly McMonagle
12/02/2021, 10:35 PM