https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-kubernetes
  • c

    Charalampos Malathounis

    12/07/2022, 7:40 AM
    Hello everyone! Is this a good place to ask for help related to prefect-kubernetes? If it is, this is my issue.
    r
    • 2
    • 4
  • a

    Anthony

    12/07/2022, 4:46 PM
    Good morning , our DevOps team set up a Prefect environment in kubernetes. Do I need the API key to be able to connect to the environment or just the PREFECT_API_URL?
    πŸ™Œ 1
    βœ… 1
    • 1
    • 1
  • a

    Anthony

    12/07/2022, 5:50 PM
    For a kubernetes install, how do I do a pip install for prefect_gitlab? Our deveops team installed the environment so I'm wondering if they need to do the install or is there a way for me to do this?
    j
    • 2
    • 9
  • j

    Josh

    12/08/2022, 10:46 PM
    I’m trying to set environment variables in a kubernetes flow run by using the job template but the environment variable value is still using the one that is set when I create my docker Image. How do I override the environment variable set when creating the image with one on a job template to dynamically change how the flow will run?
    n
    g
    • 3
    • 12
  • g

    George Coyne

    12/09/2022, 8:21 PM
    Oh man gotta knock the dust off these brain cells
    πŸ™ 1
    🧠 1
    πŸ˜‚ 1
  • g

    George Coyne

    12/09/2022, 8:21 PM
    Give me a sec to check github
  • s

    Sean Davis

    12/09/2022, 9:02 PM
    Sorry for the naive question. I'm using prefect-2.7.0 on kubernetes and agent and orion are working as expected. I now have a job that requires additional resources (eg.,
    cpu: "4"
    and
    memory: "16G"
    ). Where is the recommended place to make that adjustments in prefect to result in the final kubernetes job manifest containing those resource requests? Custom manifest? JSON patch? Is there documentation that I missed somewhere? Thanks for any pointers.
    a
    • 2
    • 3
  • j

    Josh

    12/13/2022, 2:22 AM
    Is there any way with GKE to tag logs with the prefect task id that the log belongs to? I can see flow run ID as part of the logging tabs, but can I get more granular?
    c
    • 2
    • 6
  • z

    zlee

    12/16/2022, 10:51 PM
    Hey all! I'm self-hosting prefect server via kubernetes, and the prefect-hasura component (
    <http://docker.io/hasura/graphql-engine:v2.0.9|docker.io/hasura/graphql-engine:v2.0.9>
    ) has suspiciously long
    query_execution_times
    in its logs (30-240 seconds) for some POST requests:
    {
      "type": "http-log",
      "timestamp": "2022-12-16T18:59:06.914+0000",
      "level": "info",
      "detail": {
        "operation": {
          "query_execution_time": 217.92351778,
          "user_vars": {
            "x-hasura-role": "admin"
          },
          "request_id": "0d820ccb-6294-42f1-8433-a7d1d9562684",
          "response_size": 99,
          "request_mode": "single",
          "request_read_time": 0.000004808
        },
        "request_id": "0d820ccb-6294-42f1-8433-a7d1d9562684",
        "http_info": {
          "status": 200,
          "http_version": "HTTP/1.1",
          "url": "/v1alpha1/graphql",
          "ip": "127.0.0.6",
          "method": "POST",
          "content_encoding": "gzip"
        }
      }
    }
    and it periodically says it's
    gracefully shutting down server
    , which restarts the pod. Has anyone here experienced anything like this before?
    j
    a
    • 3
    • 7
  • m

    Malek Naski

    12/19/2022, 11:04 AM
    Hello πŸ‘‹ in the agent's helm chart, it's currently not possible to specify several queues as the list of queue names will just be concatenated (with a comma in between due to https://github.com/PrefectHQ/prefect-helm/blob/f111427b1cd854e5b733f0d00c47a512a7f93006/charts/prefect-agent/templates/deployment.yaml#L55) Is this known and will it be addressed soon? Thanks πŸ™
    j
    • 2
    • 1
  • a

    aram

    12/22/2022, 1:18 PM
    hi, could someone advise on this issue?
    11:57:28.377 | INFO    | prefect.engine - Engine execution of flow run
    'ec6916a8-2d40-4ffb-9df3-cfb39f82c875' aborted by orchestrator: This run has
    already terminated
    No logs anywhere. Tasks/flows show as Crashed.
    m
    • 2
    • 2
  • j

    John LaRocque

    12/29/2022, 7:53 PM
    The orion helm chart refers to "internalPostgres" here, but I can't find this key anywhere; is it real? If not, how should I point my orion at an existing/external postgres server - will it switch automatically if I provide an externalHostName?
    j
    • 2
    • 2
  • a

    Anna Geller

    01/06/2023, 8:17 PM
    Here is a user question about network issue with prefect-helm - sharing here in case somehow knows how to help
    :gratitude-thank-you: 1
    c
    j
    +2
    • 5
    • 10
  • p

    Phil Chen

    01/10/2023, 3:42 PM
    I'm new to Prefect. Try to understand how prefect works in the context of k8s. From the documentation, looks like Prefect runs tasks as k8s jobs. Is it possible for Prefect to create any k8s CDRs. For example, if I would like to kick off a Spark run with Spark Operators's SparkApplications. How do I do that with Prefect? TIA for any help.
    j
    • 2
    • 4
  • r

    redsquare

    01/10/2023, 9:57 PM
    the path looks wrong to me should be something like "/spec/template/spec/containers/0/resources"
    :gratitude-thank-you: 2
    g
    a
    w
    • 4
    • 11
  • u

    Uday

    01/12/2023, 8:27 PM
    Hello, We have deployed prefect orion on EKS using helm chart and I am trying to start the agent locally on my laptop by setting prefect config set PREFECT_API_URL=*****, but when I try to start the agent using prefect agent start -q **, I am getting this error RuntimeError: The connection pool was closed while 1 HTTP requests/responses were still in-flight. An exception occurred. Any inputs on this are appreciated.
    n
    • 2
    • 10
  • k

    Kevin Grismore

    01/13/2023, 2:47 PM
    Hi Prefect people! I have a flow that starts around 200 subflows (all a single deployment with different parameters) as kubernetes jobs on a single work queue with a concurrency of 10. I'm running into two issues: First, a few of the subflows never start and are in either a pending or crashed state with either no logs or a single log about downloading flow code. I've increased the timeout window to 10 minutes for pods to start but that doesn't seem to help. Second, some long-running flows have their state changed to crashed with exit status -1 even though they're still running. When they reach their next task, they crash with logs about that task being already finished. Both the agent and the flows are on 2.7.7. Been stuck on this for a few days and would love some advice!
    n
    • 2
    • 7
  • s

    Shirley Monroe

    01/13/2023, 3:56 PM
    Hi folks! I'm trying to get set up to run an agent & the deployment-flows it manages in an EKS cluster. So far I can start the agent & it is monitoring the expected queue, but it turns out to be running as user
    system:serviceaccount:prefect:default
    which does not have (and should not have) the necessary permissions. How can I change that to run as a more appropriate user/role?
    c
    j
    • 3
    • 4
  • a

    Aleksandr Liadov

    01/18/2023, 2:40 PM
    Hello, In Prefect1 I can specified asked resource with
    KubernetesRun
    (
    cpu_request
    ,
    memory_request
    ,
    cpu_limit
    ,
    memory_limit
    ) In Prefect2 I use
    KubernetesJob
    , how can I specify the same parameters?
    c
    r
    j
    • 4
    • 8
  • e

    eddy davies

    01/25/2023, 12:41 PM
    Deploying my job to kubernetes with
    prefect deployment build prefect-test/main.py:main \
        -q test \
        -n test \
        -t api-to-s3 \
        -o deployment.yaml \
        -sb s3/code \
        -ib kubernetes-job/k8s-prod
    But I am getting a module not found error. I added
    "EXTRA_PIP_PACKAGES": "s3fs"
    to my Kubernetes Job block environment so that it can access code on s3 but it feels wrong to add all packages in there. I have a
    pyproject.toml
    and
    requirements.txt
    but neither seem to be being used to install packages.
    n
    j
    • 3
    • 7
  • f

    Farid

    01/30/2023, 10:01 PM
    Hi I'm getting submission errors in Kube when trying to run my flows with a custom serviceAccount:
    Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd3e45036-d667-49d4-8e3b-3a73bec788e1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2681d750-c112-4385-86a3-ed13b9c2ebf2', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f3844bc0-2265-4354-9d81-5b753f0d513e', 'Date': 'Mon, 30 Jan 2023 21:34:52 GMT', 'Content-Length': '313'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:data-eng:default\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"data-eng\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
    the deployments:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: prefect-agent
      namespace: data-eng
      labels:
        app: prefect-agent
    spec:
      selector:
        matchLabels:
          app: prefect-agent
      replicas: 1
      template:
        metadata:
          labels:
            app: prefect-agent
        spec:
          containers:
          - name: agent
            image: prefecthq/prefect:2.7.8-python3.9
            command: ["prefect", "agent", "start", "-q", "k8s-us-west-2-prod"]
            imagePullPolicy: "IfNotPresent"
            env:
              - name: PREFECT_API_URL
                value: <https://api.prefect.cloud/api/accounts/XX/workspaces/YY>
              - name: PREFECT_API_KEY
                valueFrom:
                  secretKeyRef:
                    name: prefect-cloud-api-token
                    key: prefect-cloud-api-token-value
    ---
    apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
    kind: Role
    metadata:
      namespace: data-eng
      name: prefect-agent
    rules:
    - apiGroups: [""]
      resources: ["pods", "pods/log", "pods/status"]
      verbs: ["get", "watch", "list"]
    - apiGroups: ["batch"]
      resources: ["jobs"]
      verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
    ---
    apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
    kind: RoleBinding
    metadata:
      name: prefect-agent-role-binding
      namespace: data-eng
    subjects:
    - kind: ServiceAccount
      name: prefect-agent 
      namespace: data-eng
    roleRef:
      kind: Role
      name: prefect-agent
      apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
    
    ---
    # The default manifest generated by `prefect kubernetes manifest agent` uses the default sa in the namespace.
    # We will explicitly create a sa and attach the annotation required.
    
    kind: ServiceAccount
    metadata:
      name: prefect-agent 
      namespace: data-eng
      annotations:
        <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::XX:role/an-irsa-role
    ---
    Needles to say that I have specified the same namespace and serviceAccount name in the Blocks but clearly the agent still tries to use the default serviceAccount to create jobs. any idea?
    βœ… 1
    t
    c
    • 3
    • 7
  • s

    Steph Clacksman

    02/02/2023, 12:57 PM
    I am using the helm chart to deploy orion and an agent. What would be the recommended way to create blocks and deployments, etc? I don't want to use the UI because I don't want any manual steps. Should it all go through the API?
    j
    • 2
    • 1
  • a

    aram

    02/03/2023, 1:37 PM
    hi guys, do you have resource/limit recommendation for prefect-orion and agents for example per api request? we are trying to fine tune prefect for k8s.
  • n

    nicholasnet

    02/07/2023, 4:04 PM
    Hi there
  • n

    nicholasnet

    02/07/2023, 4:05 PM
    How do you attach secret block in Kubernetes Job and how it can access those secrets during runtime?
    r
    • 2
    • 38
  • s

    Shirley Monroe

    02/07/2023, 5:00 PM
    Hi folks! I have a flow deployment with type
    kubernetes-job
    that I want to be able to enable task result caching so all the subflows within the same pod could reuse results. I think I need to set up the pods to have a volume mounted for storage; how can I specify that in the deployment config? Also, there's one task that would be called with the same parameters from multiple concurrent flows. It'd be awesome to be able to cache that result & have it available across pods, but barring that, if I set up caching for that task would it be expected to cache correctly independently for all the pods? 🀞
    c
    g
    j
    • 4
    • 17
  • z

    zlee

    02/08/2023, 5:41 PM
    I am running a flow via kubernetes job. Sometimes when there are not enough nodes available to run the job pods, it will take a sec to scale up a new node. During this time, prefect seems to mark the flow as crashed (since it was not scheduled after some time), but eventually the new node does come up and the flow is able to run fine. However, prefect refuses to run the flow since since the run has already been marked as terminated:
    aborted by orchestrator: This run has already terminated.
    Is there some way I can configure the internal timeout for waiting for the pod to be scheduled? Configuring retries does not seem to make a difference. Thanks!
    βœ… 1
    t
    • 2
    • 3
  • n

    nicholasnet

    02/16/2023, 6:41 PM
    Is there anyway to inject Vault secrets to Prefect Kubernetes job automatically without using Blocks?
    βœ… 1
    r
    j
    • 3
    • 22
  • h

    Hongbo Miao

    02/18/2023, 1:41 AM
    Hi team, do we use
    PREFECT_SERVER_UI_API_URL
    or
    PREFECT_ORION_UI_API_URL
    after the rebranding? Thanks!
    βœ… 1
    • 1
    • 2
  • h

    Hongbo Miao

    02/24/2023, 12:23 AM
    Hi everyone, has anyone met this issue before?
    ConnectionRefusedError: [Errno 111] Connect call failed (β€˜127.0.0.1’, 4200)
    I am stuck here for over a week. Any guide would be appreciate. Thanks! πŸ˜ƒ https://discourse.prefect.io/t/connectionrefusederror-errno-111-connect-call-failed-127-0-0-1-4200/2415
    • 1
    • 1
Powered by Linen
Title
h

Hongbo Miao

02/24/2023, 12:23 AM
Hi everyone, has anyone met this issue before?
ConnectionRefusedError: [Errno 111] Connect call failed (β€˜127.0.0.1’, 4200)
I am stuck here for over a week. Any guide would be appreciate. Thanks! πŸ˜ƒ https://discourse.prefect.io/t/connectionrefusederror-errno-111-connect-call-failed-127-0-0-1-4200/2415
I found the issue! I posted a workaround solution at https://discourse.prefect.io/t/connectionrefusederror-errno-111-connect-call-failed-127-0-0-1-4200/2415/3
View count: 3