https://prefect.io logo
k

ketan

08/07/2023, 1:50 PM
Hi! I've encountered a couple of issues while working with some of my flows in prefect 2.0 : 1. Issue with dbt Run and Wait_for: In some flows, I'm utilizing the map function to iterate tasks, followed by a dbt run. Surprisingly, the dbt run doesn't seem to wait for the previous tasks to complete, despite using the wait_for function. I'm unsure if I've missed something in the configuration. 2. Flow Update Recognition Problem: I made updates to one of my flows and re-deployed it. However, the flow doesn't seem to recognize the new changes and is still referencing the old configuration. I'd greatly appreciate your guidance in troubleshooting these issues.
n

Nate

08/07/2023, 5:25 PM
hey @ketan - can you share your code using
wait_for
? and on 2, can you share where your flow code lives?
k

ketan

08/08/2023, 2:21 AM
Wait_for code
2. prefect cloud
n

Nate

08/08/2023, 2:23 AM
i think for the DbtCoreOperation you actually want wait_for=write_to_s3 because the task that generates that result is being mapped, so write_to_s3 is already a list
k

ketan

08/08/2023, 2:32 AM
Thanks Nate, i will update it and check
And what's about the redeployment , its still taking the old changes
n

Nate

08/08/2023, 2:34 AM
and your flow code must be saved somewhere (prefect cloud doesnt store flow code) like github, s3 or a local file system. how did you create your deployment?
k

ketan

08/08/2023, 2:35 AM
Sorry, its local file system
n

Nate

08/08/2023, 2:37 AM
okay, how are you running your deployment when you notice that your changes are not taking effect ?
k

ketan

08/08/2023, 2:37 AM
Basically the code currently present in the agent machine
Copy code
prefect deploy --name ocrolus_analytics_flow
n

Nate

08/08/2023, 2:38 AM
okay, and then you run a worker locally and trigger a flow run from your deployment?
k

ketan

08/08/2023, 2:41 AM
I wanted to share that I am responsible for maintaining the
prefect.yaml
file, which serves as the foundation for my deployments.
Copy code
prefect deploy --name ocrolus_analytics_flow
And then using the above command i deploy the flow. Once the deployment process is complete, I'm able to execute tasks directly from the user interface (UI).
n

Nate

08/08/2023, 2:42 AM
hmm, in order to run a deployment, you must have a worker someplace thats picking up the flow runs created from your deployment (when you click quick run / custom run in the UI) - do you know where thats running / can you see its logs?
k

ketan

08/08/2023, 2:43 AM
yes, while deployment i have mentioned the worker pool, in which the flow should run
and
prefect worker start --pool 'prod-agent-1'
using this command my worker is running in the backed
n

Nate

08/08/2023, 2:55 AM
do you happen to have the worker logs? when a flow runs, the worker logs should include where its pulling the flow code from
k

ketan

08/08/2023, 2:58 AM
let me check
I have tested the
wait_for=write_to_s3
, its still not working
n

Nate

08/08/2023, 3:30 AM
hmm, okay i will look more tomorrow