https://prefect.io
Join Slack
Hi. poetry add prefect[azure] is giving error. And I am guessing this is the reason, my flows are no...
# ask-community
s
Hi. poetry add prefect[azure] is giving error. And I am guessing this is the reason, my flows are not working. Without azure they were working fine. What can be done here?
k
I don’t use poetry myself. Will find someone at Prefect to try to replicte this
s
ok
k
We can replicate. Looking into it
Ok so it does seem that 
poetry
can’t handle this nicely but pip install does. This is not an issue Prefect controls since it has to do more with poetry and the azure library. Our recommendation is to do something like
Copy code
poetry export -f requirements.txt --output requirements.txt --without-hashes
and then move to conda
s
ok, i'll give this a try
the problem is that pip install prefect[azure] is also not working for me. i have tried with the following pip versions: 21.2.4, 19.0.0, 19.3.1 is there any specific version you rcommend?
Attaching the screenshots for error with 21.2.4 (first screenshot. this keep going on for more than an hour) and 19.x (second screenshot)
k
You can just use install 
pip install azure-storage-blob
instead of the whole azure suite because they have so many packages underneath that conflict with each other
s
this is already added. but the flow is giving this error:
k
Did you restart the agent after adding it?
Is this on a local agent?
s
it is on a local agent. and yes, i have restarted agent multiple times
k
That’s very weird. Did you move to conda or using vanilla pip?
s
i tried using vanilla pip.
for my virtual env, i have added poetry add azure-storage-blob nd that works fine. however the flows are not getting executed because of the above error
k
can you run the agent with 
--show-flow-logs
and 
--log-level=DEBUG
to the agent and try running the flow?
s
sure
a
I agree that you should install it with 
pip install azure-storage-blob
to avoid dependency clashes. how far are you in the onboarding process with 1.0? if this is not working for you in 1.0, you could consider upgrading to Prefect 2.0 already 🙂 we manage dependencies more reliably in 2.0 using Prefect Collections - here is one for Azure
s
azure-storage-blob is already isntalled
a
and if you want to use 1.0 for this use case, I'd recommend trying a fresh new conda virtual environment with Prefect and then azure-storage-blob installed. Can you share your Flow object's configuration? do you use Azure for Storage only or for something else?
s
currently only for storage. we may have other uses in future, I am not sure about it yet
a
thx for sharing your Flow - I see the issue here. Since your Flow object is defined within a function, Prefect can't find it
you should move everything from the 
temp_prefect_run()
to global scope and this should fix the issue
s
i am using a POST call to run this
a
to explain why this is happening: by default Azure storage pickles your flow object and retrieves it at runtime from storage
s
how can I call the tasks if everything is not in a single function.
a
why would you want to register via a POST request?
I don't think that you have to do that. Can you explain what problem do you try to solve here? do you try to trigger a flow run via an API call?
s
we have a requirement, for which we need to create and execute flows (which will create resources in Azure using terraform). these flows need to br created and executed based on an API call
a
The thing is: every time you register your flow through 
flow.register()
or 
prefect register
CLI, you are doing it already via an API call. What exactly is the requirement? Is this about the audit trail of who registered a flow and when?
s
the requirement is - the azure resources have to be created on demand. and they can be created by users who are not well versed in azure. all they have to do is to make an PAI call, and the resources should get created.
a
I see, an interesting use case. It's up to you how you build it, but I wouldn't create a separate API for it because you can already leverage Prefect's API. Can you actually share more about your use case? (only if you can) it made me curious that some users may request Terraform-created resources - curious what is the actual problem here because it could be that you could solve your issue with a single parametrized flow and those users could just trigger those flows from Prefect UI
s
can I ping you separately about it?
a
I think we can continue here - you just don't need to share any personal/sensitive details - what do you think?
I believe if you put the arguments you supply to the Terraform class as Flow Parameter tasks, you can end up with a single flow that can be triggered with various parameters directly from the UI, which would significantly simplify your end-solution. Curious to hear your thoughts on that
if you want to learn more on various ways to trigger parametrized flow runs, check this blog post https://www.prefect.io/blog/how-to-make-your-data-pipelines-more-dynamic-using-parameters-in-prefect/
s
that sounds good Anna. let me see if I can get this working using parameterized flow
👍 1
14 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by