• bitsofinfo

    bitsofinfo

    9 months ago
    hi - new to prefect, just reading the docs and see the section on "flow storage", i guess Im trying to follow the developer workflow here... so I code a python file that defines my flow and its tasks... submit this to prefect, and it uses flow storage to actually store that script? (i,e. vs me coding a flow, checking it into git, then deploying my flows to prefect..) I guess Im not sure I'm following this.
    bitsofinfo
    Kevin Kho
    13 replies
    Copy to Clipboard
  • Jason Motley

    Jason Motley

    9 months ago
    Is there a way to set up an if/else statement within a flow depending on if a SQL query errors out? For example,
    if(select * from table) = error then (alter table add column) else somethingelse
    ?
    Jason Motley
    Kevin Kho
    8 replies
    Copy to Clipboard
  • Billy McMonagle

    Billy McMonagle

    9 months ago
    Hi there! I hope today's AWS outages have not been too stressful for the team. Due to issues in us-east-1, my prefect agents are down, and a number of flows were marked as "late". I canceled several of them manually, and then clicked "Clear late runs" in the UI. Can you clarify what this actually does?
    Billy McMonagle
    Kevin Kho
    5 replies
    Copy to Clipboard
  • Emma Rizzi

    Emma Rizzi

    9 months ago
    Hello, question on Agents to understand the doc here : https://docs.prefect.io/orchestration/agents/overview.html
    A single agent can manage many concurrent flow runs
    -> does it mean that if I schedule multiple flow at the same time for the same agent, the agent will be able to treat them parallelly ? Or does it queue the jobs ? Thanks
    Emma Rizzi
    Amanda Wee
    +1
    4 replies
    Copy to Clipboard
  • Joël Luijmes

    Joël Luijmes

    9 months ago
    Hi! I’m working on a flow which requires large concurrency (tens of thousands of tasks). Previously I’ve used the
    LocalDaskExecutor
    as my requirements for concurrency were low. However with this flow I want to use the real
    DaskExecutor
    . As I’m aware Prefect is capable of creating a temporal Dask cluster for the running flow (using
    KubeCluster
    ), but alternatively can use an existing deployed Dask cluster (or even dask gateway if I’m not mistaken). Note, I’m running in Kubernetes, so adaptive scaling could be interesting. Are there any guidelines / suggestions / experiences for using a temporal vs. static Dask cluster? Additionally, if I have
    Docker
    as storage, how can I supply the correct image tag containing my modules/dependencies while registering the flow?
    Joël Luijmes
    Anna Geller
    7 replies
    Copy to Clipboard
  • Ivan Zaikin

    Ivan Zaikin

    9 months ago
    Hi! I am trying to run
    prefect==2.0a5
    in Docker. Here is my Dockerfile:
    FROM python:3.8
    RUN adduser prefect
    USER prefect
    WORKDIR /home/prefect
    COPY --chown=prefect:prefect <http://requirements.in|requirements.in> ./
    ENV LANG C.UTF-8
    ENV LC_ALL C.UTF-8
    ENV PATH="/home/prefect/.local/bin:${PATH}"
    RUN pip install --user --no-cache-dir -r <http://requirements.in|requirements.in>
    COPY --chown=prefect:prefect flow.py flow_deployment.py ./
    Inside the container I create a deployment and several flow runs, but all of them are marked as “late”. Here is the terminal output:
    $ prefect orion start --host 0.0.0.0 --log-level DEBUG
    Starting Orion API server...
    INFO:     Started server process [71]
    INFO:     Waiting for application startup.
    09:54:06.189 | Scheduler service scheduled to start in-app
    09:54:06.189 | MarkLateRuns service scheduled to start in-app
    INFO:     Application startup complete.
    INFO:     Uvicorn running on <http://0.0.0.0:4200> (Press CTRL+C to quit)
    09:54:06.501 | Finished monitoring for late runs.
    09:54:06.538 | Scheduled 0 runs.
    Starting agent connected to <http://0.0.0.0:4200/api/>...
    Agent started! Checking for flow runs...
    09:54:07.298 | Submitting flow run 'f0855bd3-2eab-4346-ad3a-2e237a688faa'
    09:54:07.298 | Submitting flow run '521624eb-cec8-4f9b-9e92-f203e104586a'
    09:54:07.298 | Submitting flow run 'f25da7b6-7893-4107-aaa3-df22377e2ccf'
    09:54:07.299 | Submitting flow run '2895bdfa-082c-43b5-afc2-d0dcc269bf51'
    09:54:07.299 | Submitting flow run '2b445dbd-58b4-4acb-89b7-1f6782dc0ec9'
    09:54:07.300 | Completed submission of flow run 'f0855bd3-2eab-4346-ad3a-2e237a688faa'
    09:54:07.300 | Completed submission of flow run '521624eb-cec8-4f9b-9e92-f203e104586a'
    09:54:07.300 | Completed submission of flow run 'f25da7b6-7893-4107-aaa3-df22377e2ccf'
    09:54:07.300 | Completed submission of flow run '2895bdfa-082c-43b5-afc2-d0dcc269bf51'
    09:54:07.300 | Completed submission of flow run '2b445dbd-58b4-4acb-89b7-1f6782dc0ec9'
    09:54:08.969 | Flow run '521624eb-cec8-4f9b-9e92-f203e104586a' exited with exception: KeyError('__main__')
    09:54:08.975 | Flow run '2895bdfa-082c-43b5-afc2-d0dcc269bf51' exited with exception: KeyError('__main__')
    09:54:08.976 | Flow run 'f25da7b6-7893-4107-aaa3-df22377e2ccf' exited with exception: KeyError('__main__')
    09:54:08.979 | Flow run '2b445dbd-58b4-4acb-89b7-1f6782dc0ec9' exited with exception: KeyError('__main__')
    09:54:08.980 | Flow run 'f0855bd3-2eab-4346-ad3a-2e237a688faa' exited with exception: KeyError('__main__')
    Is there a way to debug these KeyErrors?
    Ivan Zaikin
    Anna Geller
    +1
    28 replies
    Copy to Clipboard
  • Ievgenii Martynenko

    Ievgenii Martynenko

    9 months ago
    Good day, Have a question about logging. We have a separate own library "LIB" that is doing some magic. This library has own root logger defined
    logger = logging.getLogger()
    . The idea is to extend Prefect Task class and run some magic using LIB library. I've read that we can add as many NAMED loggers as we want using https://docs.prefect.io/core/concepts/logging.html#extra-loggers, but since with have root one, what happens now is: Prefect logs its records as usual, but messages from LIB are not passed. I suppose this is due to LIB logger is root one. Have you ever faced such situation?
    Ievgenii Martynenko
    Anna Geller
    5 replies
    Copy to Clipboard
  • j

    Justin

    9 months ago
    Hi guys, maybe someone can point me in the right direction: I have successfully deployed Prefect to a local Kubernetes cluster, have UI & Graphql access, but I am a little lost with the deployment of Flows to the cluster... As far as I got it I need to create a Dockerimage, and then a flow using KubernetesRun, right? But how do I deploy it then?
    flow = Flow("taskname")
    
    flow.run_config = KubernetesRun(env={"POSTGRES_USER": "2234234",
                                        ...
                                    },
                                    image = "dockerimage:latest"
                                    )
    
    flow.register('projectname')
    j
    Anna Geller
    8 replies
    Copy to Clipboard
  • Vadym Dytyniak

    Vadym Dytyniak

    9 months ago
    Hi. Is it any way to pass env variable in ECSRun(env={...}) as Prefect Secret?
    Vadym Dytyniak
    Anna Geller
    +1
    31 replies
    Copy to Clipboard
  • v

    Vipul

    9 months ago
    Hi All, Orion query, I tried setting the PREFECT_ORION_API_HOST and PREFECT_ORION_API_PORT with the assumption that it would override the UI host and port. Though when I run command "prefect orion start", I can see it is starting on local host and port 4200. Am I doing something wrong?
    v
    Kevin Kho
    +1
    12 replies
    Copy to Clipboard