https://prefect.io logo
Title
s

Shuchita Tripathi

04/04/2022, 1:59 PM
Hi. poetry add prefect[azure] is giving error. And I am guessing this is the reason, my flows are not working. Without azure they were working fine. What can be done here?
k

Kevin Kho

04/04/2022, 2:28 PM
I don’t use poetry myself. Will find someone at Prefect to try to replicte this
s

Shuchita Tripathi

04/04/2022, 2:33 PM
ok
k

Kevin Kho

04/04/2022, 4:05 PM
We can replicate. Looking into it
Ok so it does seem that
poetry
can’t handle this nicely but pip install does. This is not an issue Prefect controls since it has to do more with poetry and the azure library. Our recommendation is to do something like
poetry export -f requirements.txt --output requirements.txt --without-hashes
and then move to conda
s

Shuchita Tripathi

04/04/2022, 4:17 PM
ok, i'll give this a try
the problem is that pip install prefect[azure] is also not working for me. i have tried with the following pip versions: 21.2.4, 19.0.0, 19.3.1 is there any specific version you rcommend?
Attaching the screenshots for error with 21.2.4 (first screenshot. this keep going on for more than an hour) and 19.x (second screenshot)
k

Kevin Kho

04/04/2022, 4:28 PM
You can just use install
pip install azure-storage-blob
instead of the whole azure suite because they have so many packages underneath that conflict with each other
s

Shuchita Tripathi

04/04/2022, 4:31 PM
this is already added. but the flow is giving this error:
k

Kevin Kho

04/04/2022, 4:35 PM
Did you restart the agent after adding it?
Is this on a local agent?
s

Shuchita Tripathi

04/04/2022, 4:53 PM
it is on a local agent. and yes, i have restarted agent multiple times
k

Kevin Kho

04/04/2022, 5:31 PM
That’s very weird. Did you move to conda or using vanilla pip?
s

Shuchita Tripathi

04/04/2022, 5:47 PM
i tried using vanilla pip.
for my virtual env, i have added poetry add azure-storage-blob nd that works fine. however the flows are not getting executed because of the above error
k

Kevin Kho

04/04/2022, 5:49 PM
can you run the agent with
--show-flow-logs
and
--log-level=DEBUG
to the agent and try running the flow?
s

Shuchita Tripathi

04/04/2022, 5:49 PM
sure
a

Anna Geller

04/04/2022, 5:53 PM
I agree that you should install it with
pip install azure-storage-blob
to avoid dependency clashes. how far are you in the onboarding process with 1.0? if this is not working for you in 1.0, you could consider upgrading to Prefect 2.0 already 🙂 we manage dependencies more reliably in 2.0 using Prefect Collections - here is one for Azure
s

Shuchita Tripathi

04/04/2022, 5:54 PM
azure-storage-blob is already isntalled
a

Anna Geller

04/04/2022, 5:56 PM
and if you want to use 1.0 for this use case, I'd recommend trying a fresh new conda virtual environment with Prefect and then azure-storage-blob installed. Can you share your Flow object's configuration? do you use Azure for Storage only or for something else?
s

Shuchita Tripathi

04/04/2022, 5:58 PM
currently only for storage. we may have other uses in future, I am not sure about it yet
a

Anna Geller

04/04/2022, 6:00 PM
thx for sharing your Flow - I see the issue here. Since your Flow object is defined within a function, Prefect can't find it
you should move everything from the
temp_prefect_run()
to global scope and this should fix the issue
s

Shuchita Tripathi

04/04/2022, 6:01 PM
i am using a POST call to run this
a

Anna Geller

04/04/2022, 6:02 PM
to explain why this is happening: by default Azure storage pickles your flow object and retrieves it at runtime from storage
s

Shuchita Tripathi

04/04/2022, 6:02 PM
how can I call the tasks if everything is not in a single function.
a

Anna Geller

04/04/2022, 6:03 PM
why would you want to register via a POST request?
I don't think that you have to do that. Can you explain what problem do you try to solve here? do you try to trigger a flow run via an API call?
s

Shuchita Tripathi

04/04/2022, 6:04 PM
we have a requirement, for which we need to create and execute flows (which will create resources in Azure using terraform). these flows need to br created and executed based on an API call
a

Anna Geller

04/04/2022, 6:32 PM
The thing is: every time you register your flow through
flow.register()
or
prefect register
CLI, you are doing it already via an API call. What exactly is the requirement? Is this about the audit trail of who registered a flow and when?
s

Shuchita Tripathi

04/04/2022, 6:34 PM
the requirement is - the azure resources have to be created on demand. and they can be created by users who are not well versed in azure. all they have to do is to make an PAI call, and the resources should get created.
a

Anna Geller

04/04/2022, 6:38 PM
I see, an interesting use case. It's up to you how you build it, but I wouldn't create a separate API for it because you can already leverage Prefect's API. Can you actually share more about your use case? (only if you can) it made me curious that some users may request Terraform-created resources - curious what is the actual problem here because it could be that you could solve your issue with a single parametrized flow and those users could just trigger those flows from Prefect UI
s

Shuchita Tripathi

04/04/2022, 6:39 PM
can I ping you separately about it?
a

Anna Geller

04/04/2022, 6:42 PM
I think we can continue here - you just don't need to share any personal/sensitive details - what do you think?
I believe if you put the arguments you supply to the Terraform class as Flow Parameter tasks, you can end up with a single flow that can be triggered with various parameters directly from the UI, which would significantly simplify your end-solution. Curious to hear your thoughts on that
if you want to learn more on various ways to trigger parametrized flow runs, check this blog post https://www.prefect.io/blog/how-to-make-your-data-pipelines-more-dynamic-using-parameters-in-prefect/
s

Shuchita Tripathi

04/04/2022, 7:02 PM
that sounds good Anna. let me see if I can get this working using parameterized flow
👍 1