He all, Quick question: If I have several tasks in...
# prefect-community
a
He all, Quick question: If I have several tasks inside a prefect flow, and I want the downstream tasks to wait for some upstream tasks (which are mapped), what can I do? These are not sequential flows ... to be able to use the set.upstream and wait for flow. Any help is appreciated.
1
n
Hi Andrei, not sure if may be useful the
wait_for=
on task calling.
a
I do not use Prefect 2. I tried with upstream task, but the above task is mapped. something like
task_res = call_ts.map(task_args)
new_task_res = task2(args)
new_task_res.set_upstream(task_res)
But I am not sure it works as I get this error when testing it locally`prefect.exceptions.ClientError: [{'path': ['flow'], 'message': 'Missing Authorization header in JWT authentication mode', 'extensions': {'path': '$', 'code': 'invalid-headers', 'exception': {'message': 'Missing Authorization header in JWT authentication mode'}}}]`
z
That error looks like you’re attempting to use Prefect Cloud without logging in.
❤️ 1
a
@Zanie thanks a lot!!!!! Question: how can I run prefect flows locally -> where is the cloud run configured ? I want to run the flows locally
Because i struggle to find why I can`t run these flow locally
z
How are you running the flow?
Have you looked at the
prefect run
command? The
--help
menu should be helpful 🙂
a
yep, but I took over a build that somebody else built - and they run it v ia prefect cloud and ECS. The problem is that I need to add things (new tasks) or to set upstream tasks for some other tasks. In order to not mess up Prod, I want to test these locally and I am not able to run this locally ,because each time the master flow triggers running a child flow(the flow I run is triggering other flows) -> it tried to run them on prefect cloud and it was these missing auth token issues from above. How can I cancel this cloud dependency for DEV ? I am somewhat new to infra setup for prefect
z
You cannot trigger other flows without using a backing server
You can set
prefect backend server
and run
prefect server start
to get one. You’ll need to register the flows too.
1
a
Is there a tutorial for this ?
z
@Anna Geller might have one around. There's not a great tutorial for jumping into advanced features that someone else set up 🙃 but I bet we can find some helpful resources
🙏 1
1
a
there's a lot here, not sure which problem you are trying to solve first but here are some pointers: • Troubleshooting Prefect Server issues - list of resources you may check - Deployments & DevOps - Prefect Community - for troubleshooting Server backend issues • How to Make Your Data Pipelines More Dynamic Using Parameters in Prefect - Prefect - for prefect run CLI in v1 • Topics tagged state-dependencies - for state dependencies resources
a
@Anna Geller @Zanie thanks a lot! The first issue I want to solve , it to test if the changes are made are ok (and I do not want to test it in prod :)) ) - so I would like to do it locally (hence have the master flow spin up the other flows using local compute, not using the prefect cloud )
a
why not using Prefect Cloud for development flow? as long as you don't write to some prod DB from that flow, you can run it in Cloud as well btw, you can also test your flow locally just by using:
flow.run()
in 2.0, it's even easier - I'd recommend the default version which is Prefect 2.0 docs.prefect.io if you are getting started now
a
I am taking over something that other have written. it`s not 2.0 sadly
a
gotcha, still you can totally use Cloud for development flows
a
@Zanie I still cannot schedule dependent Tasks. I want task 1 to run after flows 1,2,3 are finished, yet I am unable to do it. I scoured the whole documentation ... Do you happen to have an example or some good tutorials ?
re-birthing this....any good tutorials for scheduling dependent Tasks or flows ? I am really not managing to have an elegant solution for this 😄 @Zanie
a
did you start planning your migration to Prefect 2 @Andrei Tulbure? if you struggle to do it in v1, it makes sense to migrate it to the default version (Prefect 2), sooner or later this will need to happen so makes sense to do it now rather than building workarounds in v1 please don't tag Michael, he's doing a lot, better ask more specific question in a new thread
1
a
As I said above, I cannot sadly switch to Prefect 2.x right now ... stuck with 1.x