https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • j

    Jason Boorn

    10/22/2021, 8:22 PM
    Is it possible to run a prefect agent as a service on Windows 10? If so what's the right way to set that up?
    k
    2 replies · 2 participants
  • g

    Gabriel Milan

    10/22/2021, 8:43 PM
    is it possible to enable authentication for the UI on the helm chart deployment?
    k
    c
    13 replies · 3 participants
  • d

    David Bokor

    10/23/2021, 2:44 AM
    I’m using Helm and Terraform to deploy Prefect on Kubernetes (AWS EKS). Since the service endpoints are generated dynamically on deployment, I can’t configure the UI’s default GraphQL URL to point to the Apollo endpoint when I install the helm chart. Currently, I have to manually go the the Prefect UI and set it - as would any user. I’d rather have the setup completely automated, if possible. Any help is appreciated.
    c
    3 replies · 2 participants
  • j

    Joe Still

    10/25/2021, 1:27 PM
    Does the self-hosting include an enterprise support option?
    a
    1 reply · 2 participants
  • j

    Jason Boorn

    10/25/2021, 6:46 PM
    In a flow, I'm looking to collect the result of several different tasks, and then pump that final array of task results into another task (which basically sends the results out via email). I'm having some struggles with this - are we able to add task results to an array within a flow?
    k
    g
    100 replies · 3 participants
  • a

    Adam Lewis

    10/25/2021, 10:27 PM
    Hi, I'm deploying the version 2021.09.02 of the prefect-server helm chart (latest I believe) with ingresses enabled, and when I navigate to the address for the prefect-ui, the page is stuck with a loading bar. (see image below). The apollo url works as expected. Could you point me in the right direction as to why it's stuck loading?
    a
    3 replies · 2 participants
  • s

    Simon Gasse

    10/26/2021, 11:49 AM
    Hi! We are using Prefect with an S3 backend for a few month now and have accumulated >100GB in
    .prefect_result
    files. Is there an automated way in Prefect to clear all data older than a certain date, or do we have to rely on S3 for that?
    a
    k
    5 replies · 3 participants
  • s

    Sylvain Hazard

    10/26/2021, 12:36 PM
    Hi ! I have some questions on memory usage in Prefect. We're running our flow using Kubernetes Run with a LocalDaskExecutor. The flow maps a bunch of tasks, with a total of about 50K tasks for it to finish. What we see is that the memory usage of the runner pod steadily increases for all the duration of the flow. Whenever it reaches its predefined memory limit, we see a sudden drop in usage but the flow keeps running properly. Is it something that is to be expected from a flow with that many tasks ? Is there anything we could do to prevent using as much memory ?
    a
    k
    +1
    8 replies · 4 participants
  • o

    Oliver Catterson

    10/26/2021, 5:06 PM
    Hi, We are have a odd issue where after around 30 minutes of a flow running, the arguments passed to a task appear as NoneTypes. "`"AttributeError: 'NoneType' object has no attribute 'start_date'""` We running prefect server 0.15.6 with the KubernetesRun run config and DaskExecutor. I have verifyed that the time is the issue by implementing a delay in a upstream task - without the delay the flow run completes successfully.
    k
    h
    12 replies · 3 participants
  • h

    Harish

    10/27/2021, 3:17 AM
    Hi, How can I set Prefect limit on the number of flows an agent can run? I want to make sure the agent only runs 1 flow. I'm using the cloud version with standard option.
    k
    a
    7 replies · 3 participants
  • c

    Christopher Chong Tau Teng

    10/27/2021, 8:27 AM
    Hi, I am facing issue with connecting the UI to graphQL endpoint. I ran Prefect server using Docker in a GCP VM, which doesn’t have any public IP as it sits behind a bastion host. I can connect to UI via ssh tunnelling, however on UI it says
    Couldn't connect to *Prefect Server* at *<http://localhost:4200/graphql>*
    . It’s mentioned here that the public IP of the VM should be used, but my VM doesn’t have any public IP.
    a
    9 replies · 2 participants
  • s

    Sylvain Hazard

    10/27/2021, 10:15 AM
    Hi ! I'm trying to run flows using temporary Dask clusters as described here. Using the exact same code as in the article, it does not seem to work as intended. When launching the flow, a job pod is properly created, followed by a dask root pod. It lasts for a few seconds then the dask pod disappears and the job pod terminates. Job pod logs are below. Got any idea what might be going wrong ?
    a
    23 replies · 2 participants
  • p

    Prasanth Kothuri

    10/27/2021, 4:59 PM
    I am running my flows in DockerAgent, the image I want to use is in private gitlab registry, how can I specify username and token required to pull the image ?
    a
    k
    11 replies · 3 participants
  • j

    Jovan Sakovic

    10/28/2021, 1:29 PM
    Hi everyone 👋 I’m trying to run a Flow locally, with a local agent and can’t get it past the “Submitted for execution” state.. Tried a lot of different stuff: • shutting down the agent (though this seems like something not very well documented, please correct me if I’m wrong) • matching the labels of the agent and the flow • switching things up with the environment variables (and Secret()) In the agent logs this is the error I get. The messed up thing is that everything was running smoothly 2 days ago..
    k
    12 replies · 2 participants
  • p

    Peter Peter

    10/28/2021, 2:46 PM
    Hey, Is there any documentation on how prefect and Kubernetes work together? It is a little unclear if all flows run get submitted to Kubernetes even if there is no capacity or if prefect queues these and submits the job when capacity exists, If it is Perfect that manages this is it Kubernetes agent that deals with this? TIA
    k
    6 replies · 2 participants
  • m

    m

    10/28/2021, 3:53 PM
    was someone able to run prefect server in local with docker container or something ?
    k
    m
    51 replies · 3 participants
  • p

    Paul Schale

    10/28/2021, 11:43 PM
    I’m having trouble running prefect server on my local machine (MBP with intel processor). I believe it’s run successfully before, but today I’m getting this error message:
    Error response from daemon: Ports are not available: listen tcp 127.0.0.1:5432: bind: address already in use
    . I’ve tried restarting the computer, reinstalling docker desktop, and making a fresh conda environment with a fresh installation of prefect, but the error persists. When I run
    lsof -i:5432
    , I get no output. Any ideas for next troubleshooting steps?
    j
    5 replies · 2 participants
  • a

    Alexis Lucido

    10/29/2021, 8:36 AM
    Hi everyone. Being a current (soon to be former) user of Airflow, I have decided to move our infrastructure to Prefect, and I am just starting to test some pipelines. They're so easy to configure. However, I am wondering what would be the cleanest way to make a flow retry an upstream task if a downstream task fails. Is there something like Airflow subdags, allowing to group tasks inside a flow? Thanks a lot
    a
    4 replies · 2 participants
  • l

    Lon Nix

    10/29/2021, 2:10 PM
    Is it possible to use a github app or a sshkey auth instead of a github access token?
    k
    7 replies · 2 participants
  • s

    Sam Werbalowsky

    10/29/2021, 2:37 PM
    Hello - I am mostly deployed and things are working well - running prefect server with DaskExecutor. I have created a custom wrapper around the Flow class so that I can easily set the run environment for local development vs production flows. I have a function that is creating the dask scheduler/worker through dask gateway - all works, EXCEPT that the dask cluster is also spun up on registration. Is there an easy way to prevent this? I am wondering if I have to make a custom dask class and use that as part of the executor, rather than a custom function that creates it as part of initiating the flow.
    k
    a
    15 replies · 3 participants
  • r

    Rafael Sá

    10/29/2021, 2:55 PM
    Hello, i tried to run prefect server start but i'm getting message: ModuleNotFoundError: No module named 'packaging'. Somebody with same problem ?
    k
    a
    2 replies · 3 participants
  • p

    Prasanth Kothuri

    10/29/2021, 5:03 PM
    Hello, I am using Docker agent to run flows, one of the task inside the flow needs to interact with k8s cluster which requires KUBECONFIG, currently I am making this available with
    --volume
    flag, not sure if I am doing it correctly here, what is the best practice ?
    k
    a
    8 replies · 3 participants
  • m

    Mike Lev

    10/31/2021, 2:02 PM
    hey all I am trying to deploy flow runs with a local agent on a local server started with
    prefect server start
    any full examples of such use cases or best practices? just for additional context running into some issues while working inside of a devcontainer where access to
    localhost
    is klunky if possible is there a sample flow file that shows how to run and deploy the flow on the server backend?
    a
    6 replies · 2 participants
  • a

    Alec Koumjian

    11/01/2021, 6:31 PM
    Hello everyone! Doing some experiments to see if Prefect might be a good fit for us. I've tried running the examples with
    server
    here: https://docs.prefect.io/core/examples/overview.html#running-with-prefect-cloud-or-server But running the jobs results in
    Failed to load and execute Flow's environment: ValueError('Flow is not contained in this Storage')
    I'm running everything on a local agent to start. What obvious thing am I missing? Thanks in advance.
    k
    9 replies · 2 participants
  • l

    Lon Nix

    11/01/2021, 9:32 PM
    is there a recommended approach for using the prefect cli with multiple server installations? I have a dev and prod setup and I'd like to be able to do something like
    prefect server create-tenant --profile dev
    k
    11 replies · 2 participants
  • l

    Lon Nix

    11/02/2021, 2:41 AM
    The prefect helm chart's values.yaml files says that
    serverVersionTag
    is
    used for the apollo, graphql, and towel services
    but the graphql image comes from the hasura repo on dockerhub, not the prefecthq one so the versioning is different. Is this a mistake or am I missing something?
    a
    2 replies · 2 participants
  • l

    Lon Nix

    11/02/2021, 8:58 PM
    Any idea what this means? It's in the
    Failed flows
    section of the dashboard. I can't find any errors in pod logs
    Something went wrong while trying to fetch failed flow information. Please try refreshing your page. If this error persists, return to this page after a few minutes.
    a
    x
    +1
    23 replies · 4 participants
  • l

    Lon Nix

    11/02/2021, 11:27 PM
    Does anyone know of a prefect image that has an ssh client installed? I just need this (from official docs)
    FROM prefecthq/prefect:latest
    RUN apt update && apt install -y openssh-client
    I feel like someone else has to have made this and put it out on dockerhub already. I know, I'm being lazy. 🙂
    k
    2 replies · 2 participants
  • c

    Christopher Chong Tau Teng

    11/03/2021, 6:34 AM
    Hi, for the docker-compose.yaml generated by
    prefect server config
    , is there any way I can configure the UI service to start in other port than 8080 in the container? I don’t mean port mapping to docker host 8080:8080 here. I mean the port that UI service runs at in the container itself. Perhaps through some environment variables like UI_PORT?
    a
    6 replies · 2 participants
  • a

    Architha Rao

    11/03/2021, 11:31 AM
    Hi. I am completely new to prefect and I am facing an issue. I have a prefect docker agent deployed. A flow attached to it shall be triggered when an input is passed and the flow shall be immediately set to run state. It's been working perfectly fine until yesterday I observed the flows are in the scheduled state infinitely. SO every new input keeps adding new fllow runs but all into an infinitely scheduled state. Any inputs would be helpful. Thanks in advance.
    p
    2 replies · 2 participants
Powered by Linen
Title
a

Architha Rao

11/03/2021, 11:31 AM
Hi. I am completely new to prefect and I am facing an issue. I have a prefect docker agent deployed. A flow attached to it shall be triggered when an input is passed and the flow shall be immediately set to run state. It's been working perfectly fine until yesterday I observed the flows are in the scheduled state infinitely. SO every new input keeps adding new fllow runs but all into an infinitely scheduled state. Any inputs would be helpful. Thanks in advance.
p

Prasanth Kothuri

11/03/2021, 11:40 AM
is the agent running, if yes check the labels on the flow and agent to make sure flow can be scheduled on the agent
:upvote: 1
a

Architha Rao

11/03/2021, 12:10 PM
Hi. Yes. The agent is running and the labels on the flow and the agent match.
View count: 2