https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • b

    Baum

    09/14/2021, 10:30 AM
    Hi, I understand there are timeout for a task, do you have timeout for a flow? Also, I want to have only one flow run at a time for a flow, is this possible?
    k
    • 2
    • 4
  • a

    Alfie

    09/14/2021, 1:22 PM
    Hi Team, is there any guide about resource planning for a local Prefect agent? I run it in a Kubernetes pod with limited memory allocated. But sometimes the process get killed because of OOM and cause the flow failed to run. My flow should not take too many memory, so I’d like to know if there is any material I can refer to about memory usage of the agent and proper way to plan for it. Thanks.
    k
    • 2
    • 3
  • a

    Alexander van Eck

    09/14/2021, 2:09 PM
    Morning team! 👋 Q: When a prefect flow is using DaskExecutor and is cancelled is there a way to force the workers to not finish the task they’re working on? What I’m seeing is that dask-workers are gracefully stopped when a flow is cancelled. For long running tasks it can be quite a while before the resources are freed.
    k
    • 2
    • 2
  • r

    Raed Shabbir

    09/14/2021, 7:15 PM
    Hello, I can’t seem to find the local development guide in the docs anymore, how do I start prefect server & agent locally for the first time?
    k
    • 2
    • 10
  • r

    Raed Shabbir

    09/15/2021, 6:56 PM
    Hello, when I deploy prefect server using the instructions found here https://github.com/PrefectHQ/server/tree/master/helm/prefect-server I find the UI says
    Connected to Prefect Server at <http://localhost:4200/graphql>
    However the Apollo GraphQL API URL actually is at an external IP. How would I update this manually on the UI? The getting started tab is not available. I tried to perform a
    helm upgrade … --set ui.apolloApiUrl=http://$API_HOST:4200/graphql
    the output of this shows
    he UI has been configured to point to 'http://<EXTERNAL_IP>:4200/graphql' by default.
    which is correct, however when I refer to the UI it still says
    Connected to Prefect Server at <http://localhost:4200/graphql>
    . How can I ensure this change was made correctly ?
    k
    • 2
    • 1
  • y

    YD

    09/15/2021, 9:08 PM
    Prefect server stopped working and not responding tried
    prefect server stop
    and then
    prefect server start --detach --use-volume
    but it get stuck
    $ prefect server start --detach --use-volume
    Pulling postgres ... done
    Pulling hasura   ... done
    Pulling graphql  ... done
    Pulling towel    ... done
    Pulling apollo   ... done
    Pulling ui       ... done
    Creating network "prefect-server" with the default driver
    temp_data_processing_postgres_1 is up-to-date
    Creating temp_data_processing_hasura_1 ... done
    temp_data_processing_graphql_1 is up-to-date
    Creating temp_data_processing_towel_1  ... done
    Creating temp_data_processing_apollo_1 ... done
    Creating temp_data_processing_ui_1     ... done
    does not detach on on the UI I get
    k
    m
    z
    • 4
    • 26
  • s

    Sam Werbalowsky

    09/15/2021, 9:57 PM
    I am trying to run a simple flow to validate that I have my gitlab storage configured correctly. I’ve deployed Prefect Server using the helm chart on Kubernetes, with the agent. I am getting an error that the secret is not found…where/how should I be setting the secret? I had tried exec-ing into the agent and setting it via export but that doesn’t seem to work. That wouldn’t solve the issue of pulling secrets somewhere automatically in any case. I don’t want to set the secret in the flow, since it would be sensitive information. How can I set them via helm or a different way?
    g
    k
    c
    • 4
    • 11
  • m

    Michael Hadorn

    09/16/2021, 7:25 AM
    Does someone else, having problem with browser cache in the UI: https://github.com/PrefectHQ/ui/issues/1050
    k
    n
    • 3
    • 6
  • a

    ale

    09/16/2021, 4:12 PM
    Hey folks 👋 We are planning Prefect Server upgrade from 0.13.13 to 0.15.x We are thinking of doing a 3-stage upgrade: from 0.13.13 to 0.13.19, then from 0.13.19 to 0.14.22, then from 0.14.22 to 0.15.x However, we are wondering if we can skip the second stage and go directly from 0.13.19 to 0.15.x Any suggestions?
    m
    k
    g
    • 4
    • 31
  • e

    Elliot Oram

    09/16/2021, 5:02 PM
    Hey folks I've just upgraded to 0.15.5 and I'm now getting a Foreign key violation on starting a local prefect server. I run
    prefect server start --postgres-port 5433
    to start the server (have another postgres instance on my machine hence the port diff) then I get the following error. However despite this and the logs spitting out endless errors the I can still connect agents and register flows and they still work. Any ideas how I can fix this?
    Foreign Key Violation.txt
    z
    k
    • 3
    • 7
  • t

    Tom Baldwin

    09/17/2021, 12:29 AM
    Will Automations only be available on Prefect Cloud (not server)? Seems that way, but this documentation page doesn't display the blue "cloud" label that the other cloud-only features (like KV Store) have.
    k
    m
    +2
    • 5
    • 17
  • j

    James Geiger

    09/17/2021, 6:26 PM
    Hello folks, fairly new to Prefect and working to get things set up in a fairly restricted environment. Any tips/tricks to install prefect server/ui using Docker on a server with no internet access?
    k
    n
    • 3
    • 5
  • n

    Nivi Mukka

    09/17/2021, 8:26 PM
    Hello Team, using Prefect Server (version 0.14.1) with Dask Gateway. I keep seeing this error after building the Docker image and registering the flow. Any insights into what might be causing this?
    dask_gateway.client.GatewayClusterError: Cluster 'dask-gateway.227aa200f55a45a091b3cf6a0d037563' failed to start, see logs for more information
    k
    z
    • 3
    • 17
  • d

    Diego Alonso Roque Montoya

    09/17/2021, 11:24 PM
    Hello, we often find ourselves having to explicitly reschedule flows even when we have enough workers on our DaskKubernetes cluster. is there a common reason this happens?
    k
    • 2
    • 10
  • g

    gayle tan

    09/20/2021, 3:42 AM
    Hi, I have deployed prefect-server via Helm charts and configured a K8s agent for an on-prem server. I’m seeing that the Kubernetes has 0 submittable runs and i cant seem to trigger a quick run of the demo file. is there any additional config required when submitting/registering a flow, when i set the prefect config?
    [server]
      endpoint = "<http://myserver.name/backend>"
      [server.ui]
      endpoint = "<http://myserver.name/>"
      apollo_url= "http:/myserver.name/backend"
    [ui]
      endpoint = "<http://myserver.name>"
    k
    a
    • 3
    • 6
  • a

    Alexander van Eck

    09/20/2021, 2:00 PM
    Hello 👋 We run prefect on-prem, works well 🙌 How do we turn on automations? the docs don’t mention a config section. https://docs.prefect.io/orchestration/concepts/automations.html#automations
    b
    k
    • 3
    • 7
  • p

    Pierre Monico

    09/21/2021, 5:13 PM
    Quick question regarding the Docker Agent: I have a VM on which I run Prefect Server + a Docker Agent. Since I am using DockerRun, I need to pull the image from a registry. I have all authorizations properly set up (doing
    docker pull myregistry/myimage:latest
    from the VM works flawlessly), yet I can’t run my flows since I get a 500:
    500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=myregistry%2Fmyimage>: Internal Server Error ("Head <https://myregistry/v2/myimage/manifests/latest>: unauthorized: authentication required, visit <https://aka.ms/acr/authorization> for more information.")
    My registry is on Azure Container Registries and from the VM I am properly logged in and can pull the image. Does the agent need some sort of additional authentication?
    k
    • 2
    • 17
  • n

    Nivi Mukka

    09/21/2021, 5:32 PM
    Hi Team, using Prefect Server (version 0.14.1) with Dask Gateway. Any insights how to fix this GraphQL error? I don’t see anything on the Dask worker logs.
    k
    • 2
    • 5
  • r

    Rob Douglas

    09/21/2021, 6:06 PM
    Hi all 👋 I have a scheduled
    Flow
    (runs daily on a
    Schedule
    with a
    CronClock
    ). I'd like to generate names of a particular format for the runs. (Rather than the defaulted random word-pair names). Is that something I can do programmatically as part of the
    Flow
    object? Can I use a callable to some parameter of the flow to determine run names? Any help is appreciated!
    k
    k
    a
    • 4
    • 13
  • n

    Nivi Mukka

    09/21/2021, 6:49 PM
    Hi Team, using Prefect Server (version 0.14.1) with Dask Gateway. I am using the
    map
    method under Flow to run the same task in parallel for different inputs. It is working fine but not using all the Dask workers, which is causing Dask to run into resource/communication errors. How can I tell Prefect flow to run only one mapped task per worker?
    k
    • 2
    • 14
  • j

    Joe McDonald

    09/21/2021, 8:43 PM
    Any Apollo maintainers here? Having issues behind ELB of occasional 502 errors, this seems to be similar to the issue I encounted with graphql, where the default framework keep alive timeout is set lower than what AWS ALB ELB uses, causing connection reset issues when the timing is right… This article explains the issue and fix: https://shuheikagawa.com/blog/2019/04/25/keep-alive-timeout/ Be sure to look at the updated part at the bottom setting:
    server.keepAliveTimeout = 61 * 1000;
    server.headersTimeout = 65 * 1000; // This should be bigger than `keepAliveTimeout + your server's expected response time`
    c
    • 2
    • 5
  • n

    Nivi Mukka

    09/21/2021, 11:00 PM
    Hi Team, is the
    result
    parameter in the @task decorator for defining tasks in a flow script mandatory? I am using the
    gcs_result
    method to save the task results but running into a weird error when its a mapped task trying to write results at the same time I think. this never happened before.
    k
    • 2
    • 10
  • a

    Andreas Eisenbarth

    09/22/2021, 9:13 AM
    Hello, we have a workflow for a single dataset that we want to run in batch over many datasets. For this we are attempting to create a "super flow" (instead of starting many separate flows in a loop) which creates child flows with create_flow_run. However, the
    create_flow_run
    task always fails when we use it with
    map
    . Also, the log does not contain any exceptions or reasons for errors. • Are we doing something wrong or is it a bug? • How can we get full logs of the tasks that are run within create_flow_run? (in parent_flow_succeeding) • The prefect UI shows no runs (zero count, no run history) for parent_flow_succeeding although it was run, it only shows runs for child_flow.
    k
    • 2
    • 4
  • a

    Alexey Gorchakov

    09/22/2021, 9:44 AM
    Greetings! Our team used the prefect for 5 month and this is a very cool tool, thank you very much for it! Unfortunately, we have faced an issue that we cannot fix. It seems that we don't catch the architecture or smth else. I hope that our case will be useful for other teams using Prefect. We have an issue with running of the remotely registered Flow. [details moved to the thread]
    k
    • 2
    • 9
  • h

    Hugo Polloli

    09/22/2021, 9:46 AM
    Hi, it seems prefect-server broke on my end. Whenever I access my public EC2 IP Address, on port 8080 I get "Unable to connect". My security group is already well configured (I tried it with a simple
    python -m http.server 8080
    and it worked perfectly fine) so it seems to be an issue with prefect server, the thing is I changed absolutely nothing, when the server is running, I can do a
    wget localhost:8080
    or
    wget 0.0.0.0:8080
    and get the prefect UI html with no issue, same thing, I can register a flow with no issue. It simply seems to be a connectivity issue between the outside world and my prefect server, despite both 8080 and 4200 ports being opened on EC2
    m
    • 2
    • 6
  • g

    Gareth Allen

    09/23/2021, 7:21 AM
    I hope I’m using the right channel. I’m currently running flows from prefect cloud. The agent is installed on a EKS cluster. Jobs run a custom Docker image. When I fire a job the pods crash with the following error:
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 49, in flow_run
        result = client.graphql(query)
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 311, in graphql
        raise AuthorizationError(result["errors"])
    Any ideas?
    k
    • 2
    • 1
  • m

    Mohamed Hajji

    09/23/2021, 9:43 AM
    hello can someone help me on how i can create mutiple agents using this helm chart ? https://github.com/PrefectHQ/server/tree/master/helm/prefect-server
    d
    • 2
    • 7
  • s

    Sam Werbalowsky

    09/23/2021, 11:09 PM
    I am running into an issue importing helper files, deployed on Kubernetes, using KubernetesAgent and DaskExecutor…file structure is:
    flows
    |____my_flow.py
    |____my_tasks.py
    |___helpers.py
    I am getting
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'my_tasks'")
    when trying to run the flow in
    my_flow
    . I have a feeling that what’s happening is the
    my_tasks
    file only lives on the Agent or Job Pod, and not the Dask pods…any way around this?
    k
    • 2
    • 13
  • y

    Yury Alexandrov

    09/26/2021, 7:34 AM
    Greetings! Is there a way to pass a default run config via env variables? (ECS run)
    k
    • 2
    • 2
  • k

    kojisuganuma.dog

    09/26/2021, 6:55 PM
    Hello👋 I’m new to Prefect Cloud. I encountered that my flow was stuck in a
    Submitted
    state but I cannot find the reason. Any ideas how I can fix this? I’m using Prefect Cloud and Google Kubernetes Engine,
    KubernetesAgent
    ,
    KubernetesRun
    , GCS storage. Details are in this thread.
    ✅ 1
    k
    • 2
    • 7
Powered by Linen
Title
k

kojisuganuma.dog

09/26/2021, 6:55 PM
Hello👋 I’m new to Prefect Cloud. I encountered that my flow was stuck in a
Submitted
state but I cannot find the reason. Any ideas how I can fix this? I’m using Prefect Cloud and Google Kubernetes Engine,
KubernetesAgent
,
KubernetesRun
, GCS storage. Details are in this thread.
✅ 1
My Flow
import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun
from prefect.storage.gcs import GCS

from prefect_flows.utils.run_config import make_kubernetes_run_config

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello world!")

with Flow(FLOW_NAME) as flow:
    hello_task()

flow.run_config = KubernetesRun(
    job_template=make_kubernetes_run_config(
        run_name="example02",
        prefect_cloud_auth_token=<token>,
    ),
)

flow.storage = GCS(
    project=<project>,
    bucket=f"well-{<env>}-prefect-flow-mlops",
    key="flows/example/example02.py",
)
KubernetesAgent’s RBAC https://docs.prefect.io/orchestration/agents/kubernetes.html#rbac Kubernetes Run’s RunConfig
import uuid


def make_kubernetes_run_config(run_name: str, prefect_cloud_auth_token: str) -> str:
  run_id = uuid.uuid4()
  return f"""
apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job-{run_name}-{run_id}
  namespace: mlops-prefect-agent
  labels:
    app: prefect-job-{run_name}-{run_id}
    identifier: {run_name}-{run_id}
spec:
  template:
    metadata:
      labels:
        app: prefect-job-{run_name}-{run_id}
        identifier: {run_name}-{run_id}
    spec:
      containers:
        - name: flow
          serviceAccountName: prefect-agent
          image: prefecthq/prefect:latest-python3.8
          imagePullPolicy: Always
          command: ["/bin/sh", "-c"]
          args: ["prefect execute flow-run"]
          env:
            - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
            - name: PREFECT__CLOUD__AUTH_TOKEN
              value: {prefect_cloud_auth_token}
            - name: PREFECT__CONTEXT__FLOW_RUN_ID
              value: ""
            - name: PREFECT__CONTEXT__NAMESPACE
              value: mlops-prefect-agent
            - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
              value: "false"
            - name: PREFECT__LOGGING__LOG_TO_CLOUD
              value: "true"
            - name: PREFECT__LOGGING__LEVEL
              value: "DEBUG"
            - name: PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS
              value: "prefect.engine.cloud.CloudFlowRunner"
            - name: PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS
              value: "prefect.engine.cloud.CloudTaskRunner"
          resources:
            requests:
              cpu: "200m"
            limits:
              cpu: "200m"
      restartPolicy: Never
      imagePullSecrets:
      - name: ""
"""
KubernetesAgent logs
[2021-09-26 18:36:49,279] INFO - agent | Deploying flow run 1cb8289f-932d-4ae6-823d-a4fdfecacbde to execution environment...
[2021-09-26 18:36:49,606] INFO - agent | Completed deployment of flow run 1cb8289f-932d-4ae6-823d-a4fdfecacbde
INFO:agent:Completed deployment of flow run 1cb8289f-932d-4ae6-823d-a4fdfecacbde
k

Kevin Kho

09/27/2021, 2:03 PM
Hey @kojisuganuma.dog, how often does this happen? Does it happen all the time?
k

kojisuganuma.dog

09/27/2021, 2:39 PM
Yes, it happens all the time.
k

Kevin Kho

09/27/2021, 2:55 PM
It seems there is something wrong with the
make_kubernetes_run_config
function. I think this will be easier if you use the
KubernetesRun()
instead to define these:
RUN_CONFIG = KubernetesRun(image = "prefecthq/prefect:latest-python3.8", service_account_name = "prefect-agent")
But I think there is confusion also. The
agent
is the process that picks up and deploys flows. The RBAC you linked is for the agent (and the task_runner class and cloud API and cloud auth token). These should be for deploying the agent. but not for the Flow. The default Flow job template is very empty .
The agent is attached to a namespace and will deploy flows as jobs in that namespace
k

kojisuganuma.dog

09/27/2021, 3:42 PM
@Kevin Kho It works!!!
The agent is attached to a namespace and will deploy flows as jobs in that namespace
I misunderstood that the namespace where jobs were deployed must be specified in the custom_job_template, but it was wrong! Thank you very much!😊
k

Kevin Kho

09/27/2021, 3:44 PM
Nice! Glad you got it working 🙂
🙌 1
View count: 2