• p

    Parwez Noori

    3 months ago
    We were considering the following architecture in Azure:1. Prefect Cloud 2. Azure container instances (as execution layer) However we have some issues when running the containers on Azure. Does anybody know any good tutorials in form of articles or videos that uses the Docker Agent with Docker? Furthermore, we are also considering the scalability of this setup in the long run. I assume that most deploy their own prefect server to kubernetes together with many pods as the execution layer. However, if we wanted to start smaller due to the cost. What could be a viable solution on Azure?
    p
    Kevin Kho
    2 replies
    Copy to Clipboard
  • s

    Slackbot

    3 months ago
    This message was deleted.
    s
    1 replies
    Copy to Clipboard
  • Aaron Goebel

    Aaron Goebel

    3 months ago
    I'm triggering a flow from the cloud UI to run on kubernetes. Kubernetes agent is authenticated. The kubernetes agent is spawning jobs, but they fail with this error:
    Aaron Goebel
    Kevin Kho
    13 replies
    Copy to Clipboard
  • Domenico Di Gangi

    Domenico Di Gangi

    3 months ago
    Hi all! I am wondering what is the best practice, in prefect 2.0 orion, to trigger a flow run if a run of another flow fails. My concrete use case is the following: I would like to send a slack message as an alert in case a periodic flow (let's call it flow_A) fails. I have already set up flow_B that sends a slack message using prefect-slack collection. I just need to trigger it with some input if a run of flow_A fails
    Domenico Di Gangi
    Anna Geller
    6 replies
    Copy to Clipboard
  • m

    Marcin Grzybowski

    3 months ago
    Hi, what is the best way to run "subtasks"? I know i can just use in taskA
    taskB.run()
    but then it's treated as a normal python function (so no logging, and no other prefect's magic is available for this taskB) I managed to workaround this creating a Flow in a Task, something like this:
    @task
    def taskA(config: dict):
    #...
        with Flow() as flowB:
            taskB()
    #...
        a = flow.runB()
    it seems to work - i get logs for flowB and taskB, etc I can visualize both flows using flow.visualize() - on two separate diagrams - but it's all right for me. But is this a correct approach, or should I do this some other way?
    m
    Anna Geller
    13 replies
    Copy to Clipboard
  • Shriram Holla

    Shriram Holla

    3 months ago
    Hi, We are exploring alternative options to Airflow and were wondering if Prefect is compatible with build systems. We use bazel and some of our scripts are only visible when run through bazel. Can we configure flows to run through a build system?
    Shriram Holla
    Anna Geller
    9 replies
    Copy to Clipboard
  • m

    Marcin Grzybowski

    3 months ago
    Hi, can I generate flow diagram in Prefect 2.0 from code? In 1.0 i could use flow.visualize() to get pdf with diagram.
    m
    Anna Geller
    6 replies
    Copy to Clipboard
  • Andreas Nord

    Andreas Nord

    3 months ago
    Hi! I'm running prefect agent with Azure VM, suggestions to save cost of the idle VM (I only need like 20 minutes nightly to run the flows)? Some built in functionality of Prefect that can help boot/shutdown to match with jobs
    Andreas Nord
    Kevin Kho
    7 replies
    Copy to Clipboard
  • m

    Marcin Grzybowski

    3 months ago
    Hi! So I started playing with Prefect2.0 and here is my question: tasks / and flows are visible on radar - "normal" python functions are not. If I want to achieve "granularity" on the radar I have to use subflows? As I cannot run task from task ?
    m
    Kevin Kho
    3 replies
    Copy to Clipboard
  • m

    Marcin Grzybowski

    3 months ago
    Hello again. How can I run the same Flow parallelly? I have used code fromhttps://discourse.prefect.io/t/how-can-i-run-multiple-subflows-or-child-flows-in-parallel/96 and it works. But when I simplify it and modify to run same Flow (not different 4) then I get
    RuntimeError("The task runner is already started!")
    Modified code below:
    import asyncio
    from prefect import flow
    
    @flow
    async def subflow_1():
        print("Subflow 1 started!")
        await asyncio.sleep(1)
    
    
    @flow
    async def main_flow():
        parallel_subflows = [subflow_1(), subflow_1()]
        await asyncio.gather(*parallel_subflows)
    
    if __name__ == "__main__":
        main_flow_state = asyncio.run(main_flow())
    m
    Anna Geller
    +1
    12 replies
    Copy to Clipboard