https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • s

    Stéphan Taljaard

    03/14/2022, 10:14 AM
    Hi. Are (some) values from config.toml exposed through the GQL API? I want to confirm the current active GQL query timeout value, but don't have direct access to the server to check the config.toml file.
    a
    k
    • 3
    • 3
  • c

    Christian Nuss

    03/14/2022, 6:20 PM
    Hey there again everyone! Hopefully a quick question about doing a
    create_flow_run.run(flow_name="flow-2")
    within a task created by a
    KubernetesRun
    ... assuming i have
    with Flow("flow-1", run_config=KubernetesRun(...)):
      ... creates task that does a create_flow_run.run(flow_name="flow-2") ...
    should
    flow-2
    receive/inherit the
    run_config
    from
    flow-1
    ?
    k
    w
    a
    • 4
    • 25
  • j

    jack

    03/15/2022, 1:15 PM
    How are others handling the occasional ECS/Fargate error
    Timeout waiting for network interface provisioning to complete.
    ?
    k
    • 2
    • 6
  • s

    Sharath Chandra

    03/15/2022, 3:16 PM
    Hi I am using prefect to orchestrate my spark jobs. The spark jobs are submitted with
    spark-submit
    using prefect’s
    ShellTask
    . I have created a subclass of ShellTask to invoke the `spark-submit.`The spark jobs are running on k8s. I am facing an issue where prefect tasks are not completing and continuously running. I see this happening on the tasks which are slightly long running (> 10 mins). The master flow maps over list and orchestrates the prefect
    ShellTask
    e.g.
    K8sSparkSubmitTask.map(
      id = ["id1", "id2", "idx"]
    )
    The task starts running for
    id1
    and pod status is observed to be
    successful
    . However prefect does not move to the next task. Why are these tasks not getting completed?
    k
    d
    • 3
    • 42
  • m

    Madhup Sukoon

    03/15/2022, 3:20 PM
    Hi! Running this:
    helm pull --destination /tmp/c68c98cd-c919-46b6-9e17-924f22aa2dd3 --version 2022.01.25 --repo <https://prefecthq.github.io/server> prefecthq/prefect-server
    results in an error :
    Error: chart "prefecthq/prefect-server" version "2022.01.25" not found in <https://prefecthq.github.io/server> repository
    However, I do see
    2022.01.25
    as a version when I do
    helm search repo prefecthq/prefect-server --versions
    . Can someone please point out where I am going wrong?
    k
    • 2
    • 4
  • l

    Lukas Brower

    03/15/2022, 4:10 PM
    Hey, I was wondering if there is a mechanism to send a notification when a flow run has been executing for a certain length of time. We have a state handler that notifies us to failed tasks, but a long running flow may not necessarily change states, so I’m not sure if there is another mechanism we can use beyond state handlers.
    m
    • 2
    • 2
  • s

    Slava Shor

    03/15/2022, 5:28 PM
    Python 3.10 despite the tuple typing annotation there is still
    TypeError: Task is not iterable
    exception.
    @task(timeout=timedelta(seconds=30))
    def fetch_symbols(...) -> tuple[MyCustomType, ...]:
      ...
    k
    • 2
    • 17
  • j

    Jacob Longhurst

    03/15/2022, 7:17 PM
    I’m wondering if it’s possible for Prefect to run the same Flow at the same time with different Parameters? Let’s say i have a flow call Balance and i want to run two or more instances at the same time but with different input Parameters, is that possible?
    k
    m
    • 3
    • 30
  • d

    Darren Burton

    03/16/2022, 3:24 PM
    I upgraded to My Orion instance last night to 2.0beta with the server running. It looked like everything worked from a UI perspective. However, when I run a flow they all fail with "Crash detected! Execution was interrupted by an unexpected exception. I am thinking I should probably stop all of the Prefect process and restart the Server. I don't see a command line or an option in the UI to stop or restart the Orion Server.
    m
    k
    • 3
    • 65
  • l

    Lana Dann

    03/16/2022, 4:55 PM
    is there a way to validate flows before registering them? i’d like to test things locally like storage, etc. that would otherwise cause an error on the registering step
    k
    k
    • 3
    • 4
  • b

    Bruno Nunes

    03/17/2022, 3:22 PM
    My prefect agent stopped working and is showing: `prefect.exceptions.ClientError: Your Prefect Server instance has no tenants. Create a tenant with `prefect server create-tenant`` When I try to run
    prefect server create-tenant --name default --slug default
    I'm getting the following error:
    prefect.exceptions.ClientError: [{'message': 'Uniqueness violation.', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_tenant'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Uniqueness violation.'}}}]
    It seems that the tenant already exists. Do you know what might be the problem?
    k
    • 2
    • 6
  • r

    Ross Rochford

    03/17/2022, 5:41 PM
    Hi everyone, I'd like to try Orion, I noticed that there is some asyncio support using AnyIO. However, I don't see anything listed in the task runners documentation. Is this feature disabled for now?
    k
    m
    • 3
    • 5
  • r

    Ross Rochford

    03/17/2022, 5:56 PM
    One more question, is there a docker-compose file somewhere for Orion to get started quickly? I'd rather this than using a cli. With the previous version, I had to reverse engineer the cli to create one.
    k
    m
    • 3
    • 11
  • r

    Royzac

    03/17/2022, 8:11 PM
    Is there an architectural diagram available that showcases prefect with an aws infrastructure?
    k
    a
    • 3
    • 4
  • j

    Jacob Longhurst

    03/17/2022, 10:05 PM
    I’m attempting to grab the logger out of the prefect context. I am doing something similar to this
    @task()
    def my_task():
        import prefect
        logger = prefect.context.get('logger')
    I’ve tried it both with having the
    import prefect
    at the top level outside of the task and in the task like in that example and both times it gives me this
    TypeError: Pickling context objects is explicitly not supported. You should always access context as an attribute of the `prefect` module, as in `prefect.context`
    Anyone have any ideas why it’s failing when i try to register the flow?
    k
    a
    • 3
    • 7
  • s

    Sam Brownlow

    03/18/2022, 7:51 AM
    tl;dr How scalable is Prefect Server for scheduling concurrent runs of 10s of thousands of flows? I'm evaluating Prefect Server as a solution for managing a data pipeline. How I envisioned this working was using the Prefect framework as a computationally minimal "glue." Prefect would be the central pipe through which the state of the data flows (but not the data itself). It would hand off all computation to microservices via (a)sync requests and its only concern would be controlling how the state is pushed from service to service. Prefect seems to be a great tool for this job; I really like the API and how quickly a developer can get up and running. However, there are a couple of things that I don't yet have total clarity on. This documentation describes Server as a "highly scalable scheduler" but also says that it may start degrading at "~10-20 tasks running concurrently against a typical Server." Does the above degradation occur because the
    typical Server
    is deployed to a single node via docker-compose with a single agent, instead of being deployed via something like Helm, which horizontally scales scheduled flows across a cluster of agents? How quickly does the size of the PostgreSQL database generally grow, relative to the number of flows run? Is there any reason that regularly deleting old runs would be any more complicated than as suggested here? Are there any case studies with Prefect Server being used to run 10s of thousands of concurrent flows? I see that there used to be a Nomad Agent, are there any helpful resources for running Prefect on a nomad cluster? Thanks for any advice you are able to share. I have been diving into Prefect for only the past couple of days so greatly appreciate any referential pointers here.
    a
    • 2
    • 9
  • g

    Gabriel Milan

    03/18/2022, 1:17 PM
    Hi all! I'm having issues with my Hasura pods. It just keeps restarting every once in a while and I'm not sure about why this is happening. The values for the helm chart I'm using for Hasura are the following:
    hasura:
      image:
        name: hasura/graphql-engine
        tag: v1.3.3
        pullPolicy: IfNotPresent
        pullSecrets: []
    
      service:
        type: ClusterIP
        port: 3000
    
      labels: {}
      annotations: {}
      replicas: 2
      strategy: {}
      podSecurityContext: {}
      securityContext: {}
      env: []
      resources:
        limits:
          cpu: "500m"
          memory: "1Gi"
        requests:
          cpu: "100m"
          memory: "256Mi"
      nodeSelector: {}
      tolerations: []
      affinity: {}
    both the pods crash at the same time and keep restarting a few times before it's back up, so my Server gets a significant amount of downtime. I don't think the resources are underestimated, but I do see a pattern on RAM usage when this happens (I've sent a screenshot). I've also attached some logs for the pods, but I can't find any relevant information on them. Is there anywhere else I could gather information for this issue?
    prefect-hasura-57f8b8d4bb-zdd65.logprefect-hasura-57f8b8d4bb-68z2z.log
    k
    a
    • 3
    • 9
  • a

    Aaron Ash

    03/21/2022, 5:48 AM
    i have prefect-server (including apollo, graphql, hasura, postgres, towel and ui) and a docker agent running in a local docker-compose environment and i am able to kick off a flow run with the cli
    prefect run --name ...
    but the agent doesn't seem to actually run the flow logic. Watching the prefect agent logs i see:
    agent_1        | [2022-03-21 05:41:42,906] INFO - agent | Registering agent...
    agent_1        | [2022-03-21 05:41:42,926] INFO - agent | Registration successful!
    agent_1        | 
    agent_1        |  ____            __           _        _                    _
    agent_1        | |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
    agent_1        | | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
    agent_1        | |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
    agent_1        | |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
    agent_1        |                                            |___/
    agent_1        | 
    agent_1        | [2022-03-21 05:41:42,933] INFO - agent | Starting DockerAgent with labels ['default']
    agent_1        | [2022-03-21 05:41:42,933] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    agent_1        | [2022-03-21 05:41:42,933] INFO - agent | Waiting for flow runs...
    agent_1        | [2022-03-21 05:44:39,049] INFO - agent | Deploying flow run 71f0c5d3-b73b-4537-8267-b0fd30325fd3 to execution environment...
    agent_1        | [2022-03-21 05:44:39,082] INFO - agent | Pulling image localhost:5000/my-flow:2022-03-21t05-43-58-036580-00-00...
    agent_1        | [2022-03-21 05:44:39,284] INFO - agent | Successfully pulled image localhost:5000/my-flow:2022-03-21t05-43-58-036580-00-00
    agent_1        | [2022-03-21 05:44:39,689] INFO - agent | Completed deployment of flow run 71f0c5d3-b73b-4537-8267-b0fd30325fd3
  • a

    Aaron Ash

    03/21/2022, 5:49 AM
    Is it not actually running the flow logic or do the logs not show up on the agent?
    a
    • 2
    • 4
  • j

    Julian

    03/21/2022, 9:56 AM
    Hey all, we recently updated our prefect cluster from version 0.15.10 to major version 1.0.0. We used an existing DaskCluster to distribute our flowruns, which we could simply do by setting envs
    PREFECT__EXECUTOR__DEFAULT_CLASS
    and
    PREFECT__EXECUTOR__DASK__ADDRESS
    in our agent. Unfortunately, this approach doesn't seem to work anymore. I'm pretty sure we could initialize a DaskExecutor in our code and attach it to our flow somehow, but this would be undisireable, as it requires changing each of our existing flows, but also for each flow to come. Is it possible to achieve the same kind behaviour as before, i.e. defining the execution engine for the agent, by simply setting 1 or 2 envs?
    a
    • 2
    • 2
  • a

    Aniruddha Sengupta

    03/21/2022, 1:14 PM
    Hello all, hope that you are well. I am currently running a Prefect server inside a virtual machine along with an agent which runs flows and tasks. When I kill the server and the agent and then restart it, I see that my previous flow data has been wiped. Is there a way to ensure that this persists? Is there a way to ensure that if I kill my server and agent and then restart them, the data about flows that have been run will remain? Thanks, Aniruddha
    k
    a
    • 3
    • 3
  • b

    Bruno Nunes

    03/21/2022, 1:53 PM
    How would we retrieve the status of a flow using graphQL queries?
    k
    • 2
    • 7
  • s

    Sylvain Hazard

    03/21/2022, 2:38 PM
    [Prefect 2.0][Orion] Running through the documentation and got a couple questions. Those are about production-related patterns so please tell me if it is a bit early for these discussions. • If I understand what I'm reading correctly, Orion's
    Storage
    is quite different from Prefect 1.0's
    Storage
    as it does not only store flow code but also results. Does this mean that
    GitStorage
    is not coming back and the new recommended process is to deploy from the CI on validated changes ? Also, this is quite specific to my use case but having to connect to a cloud storage resource to run Orion on Kubernetes could be an issue. We are currently running with
    GitStorage
    and a Kubernetes-hosted PostgreSQL instance which means we can manage every Prefect component from Kubernetes configuration. • The current version of the
    KubernetesFlowRunner
    seems to use a fixed specification for the created job. Reading the code, I could not find a way to specify a template the way 1.0's
    KubernetesRun
    allows us to. Being able to create precisely sized pods to run flows has been really useful for us and I'd love to be able to keep doing it. Is it a planned feature for the future ? :thank-you: !
    a
    m
    k
    • 4
    • 4
  • l

    Lana Dann

    03/21/2022, 5:59 PM
    hey y’all can you let me know if i’m missing something? i’m building a docker image based on the prefect image
    FROM prefecthq/prefect:latest-python3.9
    but when i start up my container and try to use the prefect cli, it says prefect isn’t installed
    $ prefect
    /bin/sh: 5: prefect: not found
    k
    a
    • 3
    • 5
  • m

    Michael Aldridge

    03/21/2022, 9:59 PM
    Is there a handy dandy environment variable I can export to tell the prefect CLI that I want it to use the 'server' backend rather than 'cloud'?
    k
    c
    • 3
    • 4
  • s

    Sen

    03/22/2022, 8:50 AM
    Hi, Is there a place where I can get the sample graphql queries? I want to query all the runs for a given flow name and project name.
    s
    • 2
    • 5
  • d

    David Charles

    03/22/2022, 9:38 AM
    Hi I’m having a problem setting secrets using GQL (same with Python client):
    mutation {
      set_secret(input: { name: "FOO", value: "BAR" }) {
        success
      }
    }
    Result:
    {
      "errors": [
        {
          "message": "Cannot query field \"set_secret\" on type \"Mutation\".",
          "locations": [
            {
              "line": 2,
              "column": 3
            }
          ],
          "extensions": {
            "code": "GRAPHQL_VALIDATION_FAILED"
          }
        }
      ]
    }
    Any ideas?
    :discourse: 1
    a
    • 2
    • 4
  • d

    Daniel Nilsen

    03/22/2022, 9:39 AM
    does the helm chart support
    ssl
    ?
    👀 1
    a
    m
    • 3
    • 5
  • a

    Architha Rao

    03/23/2022, 6:41 AM
    Hi. I have prefect server set up. I see the scheduled flows are submitted bu the agent never moves it to run state. Any idea why and what the fix could be?
    p
    a
    • 3
    • 3
  • v

    Vladislav Moiseev

    03/23/2022, 9:56 AM
    Hi! Is it possible to configure a Prefect Server so that secrets can be set from the UI? I know that it’s available in the Cloud version, but for some reason I can’t use the Prefect Cloud for my task.
    a
    • 2
    • 1
Powered by Linen
Title
v

Vladislav Moiseev

03/23/2022, 9:56 AM
Hi! Is it possible to configure a Prefect Server so that secrets can be set from the UI? I know that it’s available in the Cloud version, but for some reason I can’t use the Prefect Cloud for my task.
a

Anna Geller

03/23/2022, 10:33 AM
In Server, you can only have local Secrets - check out my answer here https://prefect-community.slack.com/archives/C014Z8DPDSR/p1647942814291169?thread_ts=1647941887.823019&amp;cid=C014Z8DPDSR
View count: 7