https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • r

    Ruslan

    02/24/2021, 5:42 PM
    how to make debug level logging visible while code running localy? It is working on web ui, but it is ignored when I run code on my local machine
    c
    • 2
    • 3
  • r

    rafaqat ali

    02/25/2021, 9:45 AM
    Hi I want to deploy prefect server to the azure web app. Can anyone let me know how I can deploy? In prefect document, the only way to deploy the server is by using the command prefect server start. The use case is : I want to provide a docker compose file to client in which are the prefect services are listed. But, I don't find any mechanism to achieve this? Your help is highly required.
    m
    • 2
    • 11
  • a

    Andrew Druchenko

    02/25/2021, 5:47 PM
    Hi, we’re running our own Prefect (0.13.10) deployment on AWS using Fargate agent. Most of the services run on a single EC2 node (t2.medium) via docker-compose. From time to time, some flows randomly fail with the inability to connect to apollo service (timeout). At this point I am not sure in what direction to further look at, any guidance/help will be appreciated.
    m
    • 2
    • 27
  • r

    rafaqat ali

    02/26/2021, 1:31 PM
    Hi, When I run below command, ~/.prefect path not generated. can anyone let me know where it will be present?
    prefect backend server
    a
    m
    • 3
    • 21
  • r

    rafaqat ali

    02/26/2021, 3:39 PM
    Hi, I have deployed my backend server on a different location. How can I register a flow using the url for that server? i need to flow.register('project name', 'in remote server)
    n
    • 2
    • 9
  • b

    banditelol

    02/27/2021, 5:12 AM
    Hello, I've used prefect for single node server for 3 months now lets call it B, currently I want to deploy a new small server in different region (lets call it S) and I want S to only accountable for newwer flows and I could still monitor it from the same prefect UI (hosted in B). What configuration should I do to each server? Can anyone point me to documentation for this case? Thank You!
    a
    j
    • 3
    • 5
  • m

    Mariusz Olszewski

    02/27/2021, 10:20 AM
    Hi, i am new in linux and prefect
  • m

    Mariusz Olszewski

    02/27/2021, 10:21 AM
    command prefect server start give me error that i had unsporrted docker-compose file version
  • m

    Mariusz Olszewski

    02/27/2021, 10:21 AM
    what should i do with this?
    c
    • 2
    • 1
  • i

    Igor Kaluđer

    02/27/2021, 10:36 PM
    Hi, does anyone know how to reload config.toml? Do I need to restart the server? Thank you!
    d
    • 2
    • 3
  • d

    Diego Alonso Roque Montoya

    02/28/2021, 5:06 AM
    Is there a way to select the executor from the UI?
    n
    • 2
    • 5
  • m

    Mariusz Olszewski

    02/28/2021, 8:07 AM
    hi, I had an connection error
  • m

    Mariusz Olszewski

    02/28/2021, 8:08 AM
  • m

    Mariusz Olszewski

    02/28/2021, 8:08 AM
    what can be the reason of that?
    d
    • 2
    • 2
  • m

    Mark Dodds

    02/28/2021, 5:27 PM
    I have a prefect agent running inside a docker container that connects to the prefect cloud server. When i initially register flows to run on this agent they execute fine, but if I update the task/flow logic (without changing the method signature - ie. just adding more logging statements), rebuild the container, and execute the flow I get an error
    Failed to load and execute Flow's environment
    . This is the full error
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
    This happens every time I update the flow regardless of what changes are actually made. I'm guessing I'm doing to setup or storage wrong but not really sure where to start - any direction would be much appreciated!
    d
    • 2
    • 3
  • m

    Minakshi

    02/28/2021, 11:11 PM
    Hi team, Is there a way to specify a custom DB to be used with prefect server instead of the default postgres db? Is the prefect server able to save state of running flows on prefect server deploy/upgrades and restarts?
    d
    • 2
    • 11
  • t

    Tobias Heintz

    03/01/2021, 12:55 PM
    Hey everyone, I am looking to deploy prefect server and UI on our own infrastructure. I wasn't able to find a tutorial or any in-depth information on that. The only thing I could find were instructions on how to deploy the server to a single local node (https://docs.prefect.io/orchestration/server/deploy-local.html) but this is hardly sufficient for a (near) production use case. Are there any experiences on how to run prefect server for example on AWS? Thanks very much for pointers in that direction!
    m
    j
    • 3
    • 7
  • r

    rafaqat ali

    03/02/2021, 12:41 PM
    Hi team, I have run prefect agent local start command and it gives me error:No Agent Api token provided I'm running server and not cloud. Can any guide me, how can I generate this token for server?
    g
    • 2
    • 15
  • g

    Ganesh Kalyansundaram

    03/03/2021, 2:54 PM
    Hey everyone, I wanted to query GraphQL for all the flow runs IDs that have failed. How would I do this? Is there a Prefect Python API that does something similar or do I have to use GraphQL? Would be nice if someone could help me with the GraphQL Query 😫
    a
    j
    • 3
    • 7
  • d

    David Harrington

    03/03/2021, 7:41 PM
    Hi we are running flows via a k8s agent and we are trying to disable checkpointing. We are using a k8s yaml template that sets the environment variable
    PREFECT__FLOWS__CHECKPOINTING: false
    for the flow. Based on the logs/errors we are seeing - the flow is still trying to serialize and write results. Two questions: 1. if we disable checkpointing at the flow level should prefect core not attempt to serialize the result 2. what is the appropriate way to disable checkpointing at the flow level Relevant traceback included as a comment from when we ran a job where the env var was set to false
    j
    m
    • 3
    • 14
  • d

    Dana Merrick

    03/03/2021, 9:19 PM
    how do I set the graphql endpoint via environment variable? right now I see:
    Couldn't connect to Prefect Server at <https://localhost:4201/graphql>
    but I would like to change that
    a
    • 2
    • 3
  • d

    Dana Merrick

    03/04/2021, 8:35 PM
    i'm trying to execute flow on a k8s agent using a custom image... i have
    prefect
    installed on the custom image, but when the container starts up and tries to run
    prefect execute flow-run
    , I get this error:
    Error: No such command 'execute'.
    Any idea what I am missing? I installed prefect via conda btw
    m
    • 2
    • 17
  • m

    Mathijs Schoorl

    03/05/2021, 2:12 PM
    I'm playing around with the trigger manual_only. Which seems to work great. Just something I can not really find a good answer to: Is it possible to only ask for an approval if all upstream tasks are successful? It looks like manual_only means it will always ask for approval and keeps the flow running. I would like to have an approval task fail with the status TRIGGERFAIL if any upstream tasks are in a failed state.
    n
    • 2
    • 10
  • j

    Jonathan Wright

    03/06/2021, 11:52 AM
    I am trying to understand how Prefect deals with the concept of running a flow as a user, with respect to Cloud and Server.  I believe that I’m correct to say that Server has no concept of a user. That is: users don’t login, there are no secrets and flows running on agents have no knowledge of if they were started by a schedule or a user interaction (e.g. UI, CLI, or API client). Whereas, Cloud does authenticate users and does support secrets. However I can see no controls that allow an admin to say which secrets are accessible to which users?  The context of my question is to understand our options for protecting data in databases where our users have their own credentials, which limits access to all the data, and the secrets/credential on our Prefect Server agents would likely allow access to all data. I welcome any thoughts on this topic, thank you.
    c
    • 2
    • 1
  • a

    ale

    03/08/2021, 12:52 PM
    Hey folks, I have a question regarding Prefect backend db. We’re using Bucardo to replicate/sync Prefect server backend database. Bucardo is complaining about the following tables not having a primary key defined:
    public.log
    hdb_catalog.hdb_schema_update_event
    hdb_catalog.hdb_custom_types
    utility.traversal
    We are thinking of skipping replication/sync of these tables, but I’m not sure if they contain important data that we should replicate. Any suggestion/advice is welcome 😒imple_smile:
    j
    • 2
    • 2
  • h

    Hien Dang

    03/08/2021, 2:55 PM
    Hi, I am using prefect docker agent and facing the timeout problem of docker.py:
    UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
    . I would like to increase the timeout to 120s but could not find an option where I can pass such parameters to prefect docker agent. Currently the implementation of docker agent simply uses the default timeout of 60s for
    docker.APIClient
    . I think it makes sense to give users the ability to configure such parameters
    d
    • 2
    • 3
  • s

    Stephen Mugisha

    03/08/2021, 5:02 PM
    Hello, Noob question here, How do I execute an agent run (docker agent) on a server so that if I exit/logout of the server, the process/agent stays running/executing scheduled flows? Your help is much appreciated.
    m
    m
    • 3
    • 16
  • a

    André Bonatto

    03/08/2021, 11:06 PM
    Hi folks. When I try to create a tenant with the command "prefect server create-tenant --name default --slug default", I get 401 Client Error : Unauthorized for url: http://localhost:4200/graphql. Any idea what is causing this problem? Thank you very much!
    k
    m
    +2
    • 5
    • 43
  • h

    haf

    03/09/2021, 10:23 AM
    I was hoping to get a bit of help: in what order to run
    python
    vs
    prefect register flow
    vs triggering flows via the UI? And if I change a flow, do I register it again? Is it always that I have to change the python code when I change the runtime of a flow (by doing
    flow.run()
    or
    flow.register()
    or is this something that can be inferred by the agent running the flow?
    m
    • 2
    • 2
  • j

    Josh Greenhalgh

    03/09/2021, 1:09 PM
    Hey can anyone help me understand this;
    NAME                                     READY   STATUS    RESTARTS   AGE
    prefect-agent-c58b946f9-9r59j            1/1     Running   280        21h
    prefect-server-apollo-78c9b8cbfb-bd69r   1/1     Running   0          3d12h
    prefect-server-graphql-875f7ddc-pntjp    1/1     Running   0          3d12h
    prefect-server-hasura-7897f76bcf-mtphx   1/1     Running   0          3d12h
    prefect-server-towel-6d9c9748f4-q9mrc    1/1     Running   0          3d12h
    prefect-server-ui-55f4bcb597-mmz4c       1/1     Running   0          3d12h
    The agent consistently restarts every 5 mins or so - is this expected? If not any idea how to solve? I am using the output of
    prefect agent kubernetes install
    as the spec for the deployment
    m
    m
    • 3
    • 8
Powered by Linen
Title
j

Josh Greenhalgh

03/09/2021, 1:09 PM
Hey can anyone help me understand this;
NAME                                     READY   STATUS    RESTARTS   AGE
prefect-agent-c58b946f9-9r59j            1/1     Running   280        21h
prefect-server-apollo-78c9b8cbfb-bd69r   1/1     Running   0          3d12h
prefect-server-graphql-875f7ddc-pntjp    1/1     Running   0          3d12h
prefect-server-hasura-7897f76bcf-mtphx   1/1     Running   0          3d12h
prefect-server-towel-6d9c9748f4-q9mrc    1/1     Running   0          3d12h
prefect-server-ui-55f4bcb597-mmz4c       1/1     Running   0          3d12h
The agent consistently restarts every 5 mins or so - is this expected? If not any idea how to solve? I am using the output of
prefect agent kubernetes install
as the spec for the deployment
m

Mariia Kerimova

03/09/2021, 1:25 PM
Hello Josh! So, there are couple reasons which can trigger pod restarts. Can you provide following information: What version of Prefect are you using? Can you describe the pod and share events from the pod? (run
kubectl describe po prefect-agent-c58b946f9-9r59j
) Do you set memory limits on the agent?
j

Josh Greenhalgh

03/09/2021, 2:26 PM
version: prefecthq/prefect:0.14.11-python3.8 describe;
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Normal   Pulled     33m (x291 over 22h)     kubelet  Successfully pulled image "prefecthq/prefect:0.14.11-python3.8"
  Warning  Unhealthy  13m (x590 over 22h)     kubelet  Liveness probe failed: Get <http://10.32.1.19:8080/api/health>: dial tcp 10.32.1.19:8080: connect: connection refused
  Warning  BackOff    8m47s (x3591 over 22h)  kubelet  Back-off restarting failed container
  Normal   Pulling    3m39s (x298 over 22h)   kubelet  Pulling image "prefecthq/prefect:0.14.11-python3.8"
memory limits: nope
This is full def using terrafrom k8s provider (if it helps);
resource "kubernetes_deployment" "prefect_agent" {
  metadata {
    name      = "prefect-agent"
    namespace = kubernetes_namespace.prefect.metadata.0.name
    labels = {
      app = "prefect-agent"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        app = "prefect-agent"
      }
    }

    template {
      metadata {
        labels = {
          app = "prefect-agent"
        }
      }

      spec {
        node_selector = {
          "<http://cloud.google.com/gke-nodepool|cloud.google.com/gke-nodepool>" = google_container_node_pool.fixed_compute.name
        }
        container {
          name    = "agent"
          image   = "prefecthq/prefect:0.14.11-python3.8"
          command = ["/bin/bash", "-c"]
          args    = ["prefect agent kubernetes start"]

          env {
            name  = "PREFECT__CLOUD__API"
            value = "http://<HIDDEN>:4200/graphql"
          }

          env {
            name  = "NAMESPACE"
            value = "prefect"
          }

          env {
            name  = "PREFECT__CLOUD__AGENT__LABELS"
            value = "['prefect-agent']"
          }

          env {
            name  = "PREFECT__BACKEND"
            value = "server"
          }

          resources {
            limits = {
              cpu    = "100m"
              memory = "128Mi"
            }
          }

          liveness_probe {
            http_get {
              path = "/api/health"
              port = "8080"
            }

            initial_delay_seconds = 40
            period_seconds        = 40
            failure_threshold     = 2
          }

          image_pull_policy = "Always"
        }
      }
    }
  }
}
m

Michael Adkins

03/09/2021, 3:01 PM
I believe you need to set the local address for the health check server to run ie https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml#L68-L69
j

Josh Greenhalgh

03/09/2021, 3:54 PM
hmmm - ok I removed that since it appeared to be related to
cloud
version?
Thanks!
m

Michael Adkins

03/09/2021, 4:00 PM
There are a few instances where
CLOUD
settings are just "backend" settings that we didn't have a better name for when Server was split out.
j

Josh Greenhalgh

03/09/2021, 4:04 PM
Ok so the issue is that that env var is required to set up some endpoint that the healthcheck probes? In my case there is nothing there so it keeps failing?
View count: 3