Thread
#prefect-community
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    Hi. poetry add prefect[azure] is giving error. And I am guessing this is the reason, my flows are not working. Without azure they were working fine. What can be done here?
    Kevin Kho

    Kevin Kho

    5 months ago
    I don’t use poetry myself. Will find someone at Prefect to try to replicte this
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    ok
    Kevin Kho

    Kevin Kho

    5 months ago
    We can replicate. Looking into it
    Ok so it does seem that
    poetry
    can’t handle this nicely but pip install does. This is not an issue Prefect controls since it has to do more with poetry and the azure library. Our recommendation is to do something like
    poetry export -f requirements.txt --output requirements.txt --without-hashes
    and then move to conda
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    ok, i'll give this a try
    the problem is that pip install prefect[azure] is also not working for me. i have tried with the following pip versions: 21.2.4, 19.0.0, 19.3.1 is there any specific version you rcommend?
    Attaching the screenshots for error with 21.2.4 (first screenshot. this keep going on for more than an hour) and 19.x (second screenshot)
    Kevin Kho

    Kevin Kho

    5 months ago
    You can just use install
    pip install azure-storage-blob
    instead of the whole azure suite because they have so many packages underneath that conflict with each other
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    this is already added. but the flow is giving this error:
    Kevin Kho

    Kevin Kho

    5 months ago
    Did you restart the agent after adding it?
    Is this on a local agent?
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    it is on a local agent. and yes, i have restarted agent multiple times
    Kevin Kho

    Kevin Kho

    5 months ago
    That’s very weird. Did you move to conda or using vanilla pip?
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    i tried using vanilla pip.
    for my virtual env, i have added poetry add azure-storage-blob nd that works fine. however the flows are not getting executed because of the above error
    Kevin Kho

    Kevin Kho

    5 months ago
    can you run the agent with
    --show-flow-logs
    and
    --log-level=DEBUG
    to the agent and try running the flow?
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    sure
    Anna Geller

    Anna Geller

    5 months ago
    I agree that you should install it with
    pip install azure-storage-blob
    to avoid dependency clashes. how far are you in the onboarding process with 1.0? if this is not working for you in 1.0, you could consider upgrading to Prefect 2.0 already 🙂 we manage dependencies more reliably in 2.0 using Prefect Collections - here is one for Azure
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    azure-storage-blob is already isntalled
    Anna Geller

    Anna Geller

    5 months ago
    and if you want to use 1.0 for this use case, I'd recommend trying a fresh new conda virtual environment with Prefect and then azure-storage-blob installed. Can you share your Flow object's configuration? do you use Azure for Storage only or for something else?
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    currently only for storage. we may have other uses in future, I am not sure about it yet
    Anna Geller

    Anna Geller

    5 months ago
    thx for sharing your Flow - I see the issue here. Since your Flow object is defined within a function, Prefect can't find it
    you should move everything from the
    temp_prefect_run()
    to global scope and this should fix the issue
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    i am using a POST call to run this
    Anna Geller

    Anna Geller

    5 months ago
    to explain why this is happening: by default Azure storage pickles your flow object and retrieves it at runtime from storage
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    how can I call the tasks if everything is not in a single function.
    Anna Geller

    Anna Geller

    5 months ago
    why would you want to register via a POST request?
    I don't think that you have to do that. Can you explain what problem do you try to solve here? do you try to trigger a flow run via an API call?
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    we have a requirement, for which we need to create and execute flows (which will create resources in Azure using terraform). these flows need to br created and executed based on an API call
    Anna Geller

    Anna Geller

    5 months ago
    The thing is: every time you register your flow through
    flow.register()
    or
    prefect register
    CLI, you are doing it already via an API call. What exactly is the requirement? Is this about the audit trail of who registered a flow and when?
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    the requirement is - the azure resources have to be created on demand. and they can be created by users who are not well versed in azure. all they have to do is to make an PAI call, and the resources should get created.
    Anna Geller

    Anna Geller

    5 months ago
    I see, an interesting use case. It's up to you how you build it, but I wouldn't create a separate API for it because you can already leverage Prefect's API. Can you actually share more about your use case? (only if you can) it made me curious that some users may request Terraform-created resources - curious what is the actual problem here because it could be that you could solve your issue with a single parametrized flow and those users could just trigger those flows from Prefect UI
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    can I ping you separately about it?
    Anna Geller

    Anna Geller

    5 months ago
    I think we can continue here - you just don't need to share any personal/sensitive details - what do you think?
    I believe if you put the arguments you supply to the Terraform class as Flow Parameter tasks, you can end up with a single flow that can be triggered with various parameters directly from the UI, which would significantly simplify your end-solution. Curious to hear your thoughts on that
    if you want to learn more on various ways to trigger parametrized flow runs, check this blog post https://www.prefect.io/blog/how-to-make-your-data-pipelines-more-dynamic-using-parameters-in-prefect/
    Shuchita Tripathi

    Shuchita Tripathi

    5 months ago
    that sounds good Anna. let me see if I can get this working using parameterized flow