https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • p

    Paul Gierz

    03/07/2022, 5:22 PM
    Hi all, Is there a way to start a local agent with the LocalDask executor? To clarify: from the CLI Thanks!
    k
    • 2
    • 8
  • s

    Shuchita Tripathi

    03/07/2022, 6:29 PM
    i have installed prefect server on a VM in Azure. my code (flow) is in my local machine. can i register that flow with the prefect server running in VM? If yes, can you please let me know the steps? I have once completed the flow register in my local prefect server, but now installing on cloud.
    k
    • 2
    • 6
  • c

    Christian Nuss

    03/07/2022, 6:55 PM
    Question for y'all! Can a
    @task
    be orchestrated as a job/pod on Kube?
    k
    a
    • 3
    • 3
  • s

    Shuchita Tripathi

    03/07/2022, 10:19 PM
    i have registered some flows in a remote prefect server. is there a way i can call to run these flows from some other code
    k
    • 2
    • 2
  • a

    Austin Vecchio

    03/08/2022, 1:29 AM
    I have installed prefect 1.0.0 and was previously running 0.15.2. For both versions I am getting the following error when trying to run my workflow via the following:
    python3 workflow.py
    My workflow only has one import:
    import prefect
    Any thoughts on why this might be? Any assistance would be appreciated
    k
    • 2
    • 8
  • s

    Saurabh Indoria

    03/08/2022, 8:11 AM
    Hi All, The prefect tasks which are in resume state are not going through, they remain in resume state. This is affecting our production flow runs as well. Can someone please check if there's any issue with prefect cloud? cc: @Kevin Kho @Christina Lopez @Anna Geller
    k
    c
    • 3
    • 30
  • p

    Paul Gierz

    03/08/2022, 10:45 AM
    Hello, I was curious if there was any experience or documentation about self hosting I could look at?
    k
    • 2
    • 5
  • a

    Aniruddha Sengupta

    03/08/2022, 1:55 PM
    Hello Hope that you are well. I have a parent task I have created and I am able to create a create a series of task runs based on the inputs according to this documentation: https://docs.prefect.io/core/idioms/task-run-names.html. How can I ensure that the children tasks that are created from this are able to run in parallel instead of sequentially? Do I need to set the flow's executor to be the LocalDaskExecutor? Thanks for your help,
    k
    • 2
    • 5
  • d

    Donnchadh McAuliffe

    03/08/2022, 2:55 PM
    hey everyone, is there any documentation on how to run prefect orion server on docker locally?
    k
    m
    • 3
    • 3
  • s

    Shuchita Tripathi

    03/08/2022, 5:07 PM
    I am trying to install and run prefect server in Azure Container. This is my docker file. Without the docker run command, I am getting docker compose down error. with the docker run, the error states - executor failed running
    FROM prefecthq/prefect:latest
    
    WORKDIR /app
    
    RUN apt-get -y update && \
        apt-get install -y <http://docker.io|docker.io> && \
        apt-get install -y docker-compose
    
    RUN docker run --privileged -d docker:dind
    
    RUN docker --version
    
    COPY docker-compose.yml .
    
    RUN prefect backend server && \
        prefect server start
    ```
    k
    • 2
    • 7
  • d

    Daniel Komisar

    03/08/2022, 5:32 PM
    Hello everyone, are there any rate limits on the GraphQL API? Thanks.
    k
    • 2
    • 1
  • l

    Lana Dann

    03/08/2022, 6:59 PM
    i have tasks that call operators/connectors that i’ve defined with their own loggers. but even when i set
    log_stdout=True
    those logs don’t show. any advice on how to include logs from those other loggers?
    k
    • 2
    • 3
  • s

    Shuchita Tripathi

    03/08/2022, 9:50 PM
    Can we apply rbac in prefect server or only on prefect cloud?
    k
    s
    • 3
    • 3
  • y

    YD

    03/08/2022, 11:37 PM
    Error for some users on "hello-world" code we have an agent running on a host, on user group "a" I am in this user group and can run the hello-world code without a problem. another user on that host, who is on a different user group, tries to run the same "hello-world" code he can run it using the
    flow.run()
    and it runs OK he can register it and see the flow in the dashboard, but it fails with
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/<user_name>'",)
    any thoughts? does every user group needs a different agent ?
    d
    k
    • 3
    • 3
  • k

    KhTan

    03/09/2022, 12:41 AM
    Hi! I was trying to set up a parent flow. It shows run succeeded, which doesn’t appear to be the case, as one of the flows is supposed to write to a database and nothing was written. however, if running one flow (a part of the parent flow), it does write to db. what could have gone wrong here? thanks!
    k
    a
    • 3
    • 14
  • t

    Tony Liberato

    03/09/2022, 2:00 PM
    Good morning, Prefectionists! We are sailing. I have successfully deployed two very simple Workflows about a month ago and everything is working beautifully. I'm a DBA for a small non-profit and we handle a ton of data. We decided to implement prefect so I can automate many of the data importing and ETL tasks I perform manually. I wanted to share a couple of things Prefect taught me. 1:Register the flow on the agent its intended to run on. for some reason this resolved all the import statements correctly. 2:Translating your workflow into prefect code helps to refactor the steps of the flow. I would do these in SSIS, and SSIS made the "theory" of the flow easy, but it didn't truly represent the logic of the flow. In doing so, I developed a much more concise workflow, which had a few more tasks. Let this happen: its a good thing. 3: Python is amazing. And since Prefect doesn't get in the way of that, prefect is amazing. It quite literally can do anything. 4: Docker flows: This can get messy. I found about 52 containers that were left over from all my flows being run. This is a really intelligent solution regardless, it just leave a trail. I'm sure there is a setting to automatically remove these that I just haven't found. Don't be alarmed if you have clutter in your docker instance. It well worth it. 5: When deciding to implement a flow in Prefect, you are forced to design the whole thing. No more "Oh, we'll deal with that when we get there..." This is a steep hill to climb, but you once you do, its a never-ending coast down the other side. This forces questions to the beginning of the process and helps get solid answers for them right away. Its possible that a workflow may not be a good candidate for Prefect, or maybe parts of the workflows are candidates. You actually save a lot of time designing end to end. Thanks for reading.
    :marvin: 4
    a
    k
    • 3
    • 4
  • l

    Liam England

    03/09/2022, 2:38 PM
    Hi folks, Wondering if there's a simple way to register flows remotely when using prefect server? (no cloud option in this environment, unfortunately)
    k
    • 2
    • 2
  • a

    Alex F

    03/09/2022, 4:58 PM
    Hi All , I am having issue with prefect server. Then server is starting on server ip and config.toml is set to apollo_url="http://10.1.x.x:4200/graphql" everything starts normally and no errors during start , the ui shows green connection to API, however 1-2 minutes later it turns yellow and stops working , i turn debug=true but I am not able to find logs are written or how to troubleshoot this problem. Any help would be appreciated (this is local install on ubuntu)
    k
    a
    • 3
    • 22
  • a

    Alex F

    03/09/2022, 7:56 PM
    prefect register --project "Project Name" --path ./Prefect/ I am getting error after pointing my install from localhost to 10.1.x.x IP , here is my configuration [server] host = "http://10.1.xx" port = "4200" host_ip = "10.1.x.x" host_port = "4200" endpoint = "${server.host}:${server.port}" [server.ui] host = "http://10.1.x.x" port = "8080" host_port = "8080" host_ip = "10.1.x.x" apollo_url="http://10.1.x.x:4200/graphql" raise InvalidSchema("No connection adapters were found for {!r}".format(url)) requests.exceptions.InvalidSchema: No connection adapters were found for '10.1.x.x:4200'
    k
    • 2
    • 2
  • s

    Shuchita Tripathi

    03/09/2022, 8:01 PM
    to run a flow frrom any code, we need to know the flow_id. how can we get the flow id programmatically?
    :discourse: 1
    k
    k
    +2
    • 5
    • 12
  • d

    Devin Flake

    03/09/2022, 11:36 PM
    Is there a way to get more debug info from a flow run? I'm trying to use git storage and am getting this error:
    Failed to load and execute flow run: NotGitRepository()
    m
    k
    • 3
    • 19
  • a

    Aaron Ash

    03/10/2022, 5:05 AM
    is there a way to get
    prefect server create-tenant ...
    to use a different apollo endpoint via environment variables or cli args? It defaults to
    <http://localhost:4200/graphql>
    but I'm attempting to run this in a docker-compose file and need to use a different hostname than localhost
    k
    • 2
    • 5
  • a

    Aaron Ash

    03/10/2022, 7:14 AM
    is it possible to programmatically create a project (within a tenant) with the cli?
    a
    • 2
    • 2
  • t

    Tomer Cagan

    03/10/2022, 10:45 AM
    Hi, There's a small issue in the helm chart during upgrade - it logs an incorrect message to the screen. (Details inside)
    a
    m
    • 3
    • 5
  • s

    Slava Shor

    03/10/2022, 1:24 PM
    Is it possible to define async tasks in Python using
    async def...
    ?
    s
    a
    +2
    • 5
    • 8
  • c

    Christian Nuss

    03/10/2022, 10:31 PM
    Hey all! if a
    state_handler
    is being used on a
    Flow
    , and a
    @task
    has a
    default_exception
    and the
    new_state
    is
    Failed
    ... is it possible to get that Exception for inspection in the state handler?
    k
    • 2
    • 2
  • l

    Liam England

    03/11/2022, 6:12 PM
    Hi, Wondering what the best practice would be for making secrets available to flow runs picked up by Docker Agents? Specifically looking at the
    SLACK_WEBHOOK_URL
    at the moment and have tried setting it as an environment variable for the agent, as well as adding it to the config.toml on the machine hosting prefect server but I'm still seeing
    ValueError: Local Secret "SLACK_WEBHOOK_URL" was not found.
    a
    • 2
    • 2
  • m

    madhav

    03/11/2022, 10:17 PM
    Hi all, I have a task that returns a dataframe, but instead it's returning a FunctionTask. Could it be something with a task definition?
    k
    • 2
    • 6
  • s

    Saurabh Indoria

    03/14/2022, 6:39 AM
    Hi all, There seems to be some issue with Prefect Cloud heartbeats. Randomly, some mapped tasks show an error:
    No heartbeat detected from the remote task; retrying the run.This will be retry 1 of 3.
    and then it never actually retries.. I believe the Lazarus process must kick in every 10 minutes and reschedule the task, right? CC: @Christina Lopez @Kevin Kho @Anna Geller
    a
    k
    c
    • 4
    • 21
  • m

    Martin Durkac

    03/14/2022, 8:38 AM
    Hi all, We have a running Prefect server (version 0.15.4) on EC2 (8 CPU, 64 GB of RAM) with approx. 15 docker container with specified flows. Each one docker container contains at least 1 running flow (max. 3). Every container has a local agent which is connected via --api parameter to prefect server. We have 6 flows which runs as infinite flow (when flow is finished state handlers starts same flow with create_flow_run() function). The rest of flows are scheduler to only run once an hour. The problem is that once a week or two most of our infinity flows fails on error:
    Failed to retrieve task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='apollo', port=4200): Read timed out. (read timeout=15)"))
    Sometimes the flow can continue without any problem, because we have infinite flow set to continue even when flow failed:
    def never_ending_state_handler(obj, old_state, new_state):
    if (old_state.is_running() and new_state.is_failed()):
    send_message_to_email(obj)
    if (old_state.is_running() and new_state.is_successful()) or (old_state.is_running() and new_state.is_failed()):
    time.sleep(5)
    create_flow_run.run(flow_name="our_flow", project_name = "our_project", run_name = str(uuid.uuid4()))
    return new_state
    But when we receive error:
    A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
    we are not able to continue... The flow is failed but state handler is not working anymore to reschedule failed flow. Does anybody know what may cause problem with Read timed out or Lazarus?
    👀 1
    a
    • 2
    • 3
Powered by Linen
Title
m

Martin Durkac

03/14/2022, 8:38 AM
Hi all, We have a running Prefect server (version 0.15.4) on EC2 (8 CPU, 64 GB of RAM) with approx. 15 docker container with specified flows. Each one docker container contains at least 1 running flow (max. 3). Every container has a local agent which is connected via --api parameter to prefect server. We have 6 flows which runs as infinite flow (when flow is finished state handlers starts same flow with create_flow_run() function). The rest of flows are scheduler to only run once an hour. The problem is that once a week or two most of our infinity flows fails on error:
Failed to retrieve task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='apollo', port=4200): Read timed out. (read timeout=15)"))
Sometimes the flow can continue without any problem, because we have infinite flow set to continue even when flow failed:
def never_ending_state_handler(obj, old_state, new_state):
if (old_state.is_running() and new_state.is_failed()):
send_message_to_email(obj)
if (old_state.is_running() and new_state.is_successful()) or (old_state.is_running() and new_state.is_failed()):
time.sleep(5)
create_flow_run.run(flow_name="our_flow", project_name = "our_project", run_name = str(uuid.uuid4()))
return new_state
But when we receive error:
A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
we are not able to continue... The flow is failed but state handler is not working anymore to reschedule failed flow. Does anybody know what may cause problem with Read timed out or Lazarus?
👀 1
a

Anna Geller

03/14/2022, 10:36 AM
Every container has a local agent
This is quite unusual setup - can you explain more about why you did it that way? Normally you would just spin up a docker agent on your VM:
prefect agent docker start --label yourlabel
and then Prefect agent will take care of spinning up containers for the flow run. This could be part of a problem especially because your Server is running within a container itself and the networking between containers becomes quite complicated here. With such long-running jobs you may try setting this env variable:
from prefect.run_configs import UniversalRun
flow.run_config = UniversalRun(env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"})
Also, as a workaround, you could disable Lazarus for such flow as described here. But if you want to find the root cause of the issue: do you happen to have some unclosed DB connections or other resources like HTTP clients that you use in your flow? I saw a similar issue occurring due to resources failing to close/shut down.
m

Martin Durkac

03/14/2022, 12:15 PM
First of all, thank your for your response. Our setup is too complicated, because we want to separate flows by requirements. Some flows need access to MongoDB and others need access to OracleDB. Because Oracle library for Python only works when Oracle instant client is installed, we decided to separate our flows to container to prevent messy environment in EC2. Those containers have infinite flows. We were not able to schedule them to run every 1 minute, because sometimes flow lasts only 10 seconds and sometimes when more data is processed takes up to 2-10 minutes. You suggested to use docker agent which spins up container to run flow. Does this solution decrease overall execution time? Execution takes 20 sec and what about booting up container and destroying?
a

Anna Geller

03/14/2022, 12:36 PM
I don't have benchmarks about it but it seems that using Docker agent could make your setup less "complicated" 😄 You could e.g. build one docker image with MongoDB dependencies and another one with Oracle dependencies and provide a specific image to your
DockerRun
run configuration:
flow.run_config = DockerRun(image="prefect-oracle:latest")
And as long as you use the same image, your Docker client should be able to reuse it without having to pull the image every time at runtime - I didn't benchmark this though.
✅ 1
View count: 1