Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Andrei Tulbure
10/04/2022, 4:14 PMHe all, Quick question: If I have several tasks inside a prefect flow, and I want the downstream tasks to wait for some upstream tasks (which are mapped), what can I do? These are not sequential flows ... to be able to use the set.upstream and wait for flow. Any help is appreciated.
✅ 1
n
Nicolas Felipe
10/04/2022, 4:50 PMHi Andrei, not sure if may be useful the
wait_for=a
Andrei Tulbure
10/04/2022, 4:52 PMI do not use Prefect 2. I tried with upstream task, but the above task is mapped. something like
task_res = call_ts.map(task_args)new_task_res = task2(args)new_task_res.set_upstream(task_res)But I am not sure it works as I get this error when testing it locally`prefect.exceptions.ClientError: [{'path': ['flow'], 'message': 'Missing Authorization header in JWT authentication mode', 'extensions': {'path': '$', 'code': 'invalid-headers', 'exception': {'message': 'Missing Authorization header in JWT authentication mode'}}}]`
z
Zanie
10/04/2022, 6:37 PMThat error looks like you’re attempting to use Prefect Cloud without logging in.
❤️ 1
a
Andrei Tulbure
10/05/2022, 4:47 PM@Zanie thanks a lot!!!!! Question: how can I run prefect flows locally -> where is the cloud run configured ? I want to run the flows locally
Because i struggle to find why I can`t run these flow locally
z
Zanie
10/05/2022, 5:21 PMHow are you running the flow?
Have you looked at the
prefect run--helpa
Andrei Tulbure
10/05/2022, 5:27 PMyep, but I took over a build that somebody else built - and they run it v ia prefect cloud and ECS. The problem is that I need to add things (new tasks) or to set upstream tasks for some other tasks. In order to not mess up Prod, I want to test these locally and I am not able to run this locally ,because each time the master flow triggers running a child flow(the flow I run is triggering other flows) -> it tried to run them on prefect cloud and it was these missing auth token issues from above. How can I cancel this cloud dependency for DEV ?
I am somewhat new to infra setup for prefect
z
Zanie
10/05/2022, 5:41 PMYou cannot trigger other flows without using a backing server
You can set
prefect backend serverprefect server start✅ 1
a
Andrei Tulbure
10/05/2022, 9:54 PMIs there a tutorial for this ?
z
Zanie
10/05/2022, 10:30 PM@Anna Geller might have one around. There's not a great tutorial for jumping into advanced features that someone else set up 🙃 but I bet we can find some helpful resources
:thank-you: 1
✅ 1
a
Anna Geller
10/05/2022, 10:53 PMthere's a lot here, not sure which problem you are trying to solve first but here are some pointers:
• Troubleshooting Prefect Server issues - list of resources you may check - Deployments & DevOps - Prefect Community - for troubleshooting Server backend issues
• How to Make Your Data Pipelines More Dynamic Using Parameters in Prefect - Prefect - for prefect run CLI in v1
• Topics tagged state-dependencies - for state dependencies resources
a
Andrei Tulbure
10/06/2022, 9:24 AM@Anna Geller @Zanie thanks a lot! The first issue I want to solve , it to test if the changes are made are ok (and I do not want to test it in prod :)) ) - so I would like to do it locally (hence have the master flow spin up the other flows using local compute, not using the prefect cloud )
a
Anna Geller
10/06/2022, 9:28 AMwhy not using Prefect Cloud for development flow? as long as you don't write to some prod DB from that flow, you can run it in Cloud as well
btw, you can also test your flow locally just by using:
flow.run()a
Andrei Tulbure
10/06/2022, 12:58 PMI am taking over something that other have written. it`s not 2.0 sadly
a
Anna Geller
10/06/2022, 1:52 PMgotcha, still you can totally use Cloud for development flows
a
Andrei Tulbure
10/11/2022, 8:45 PM@Zanie I still cannot schedule dependent Tasks. I want task 1 to run after flows 1,2,3 are finished, yet I am unable to do it. I scoured the whole documentation ... Do you happen to have an example or some good tutorials ?
re-birthing this....any good tutorials for scheduling dependent Tasks or flows ? I am really not managing to have an elegant solution for this 😄 @Zanie
a
Anna Geller
10/26/2022, 12:10 PMdid you start planning your migration to Prefect 2 @Andrei Tulbure? if you struggle to do it in v1, it makes sense to migrate it to the default version (Prefect 2), sooner or later this will need to happen so makes sense to do it now rather than building workarounds in v1
please don't tag Michael, he's doing a lot, better ask more specific question in a new thread
✅ 1
a
Andrei Tulbure
10/26/2022, 1:27 PMAs I said above, I cannot sadly switch to Prefect 2.x right now ... stuck with 1.x