https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • m

    Maria

    04/05/2022, 2:12 PM
    Hi, when I run Prefect server with my custom docker image, I get the following error
    Failed to load and execute flow run: ModuleNotFoundError("No module named 'pandas'")
    I don’t think it’s coming from missing pandas module. When I log into my docker image, I have no problem importing pandas. I’ve installed all the required packages in the base environment. I’ve also tried loading all the packages in the conda environment and that the conda environment is loaded upon docker image is loaded. Is this some python path issue? What I don’t understand is that i’ve done this several times before and i didn’t have this issue until now
    a
    k
    • 3
    • 27
  • l

    Lana Dann

    04/05/2022, 11:02 PM
    is it possible to set a context variable on the flow level? for context, i want to set a flow owner to the flow and then use that value in my slack state handler. but even when i set
    with prefect.context(dict(flow_owner="test")):
    before defining the flow, i still get an error
    Traceback (most recent call last):
      File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
        new_state = self.call_runner_target_handlers(old_state, new_state)
      File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
        new_state = handler(self.task, old_state, new_state) or new_state
      File "/Users/lanadann/prefect/data_prefect/lib/notifiers.py", line 13, in post_to_slack_on_failure
        f"@{prefect.context.flow_owner} "
    AttributeError: 'Context' object has no attribute 'flow_owner'
    :discourse: 1
    a
    k
    • 3
    • 30
  • k

    Konstantin

    04/06/2022, 2:39 PM
    Hi, team Prefect!
    Failed to set task state with error: ClientError([{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
        state = self.client.set_task_run_state(
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1839, in set_task_run_state
        result = self.graphql(
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 563, in graphql
        raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}]
    There is a problem with launching tasks in Prefect, on the screenshots, the last two unsuccessful launches returned an error (described above), the last launch was performed manually, ended in success.
    👋 2
    k
    a
    • 3
    • 38
  • a

    Ayah Safeen

    04/07/2022, 8:39 AM
    Hello All, Is there a way to change the prefect server port ?
    t
    • 2
    • 2
  • c

    Christian Nuss

    04/07/2022, 4:26 PM
    hey all! is there a way to have
    stdout
    generated inside a container from a
    DockerRun
    or
    KubernetesRun
    be provided back into Prefect following the successful execution of the flow?
    k
    • 2
    • 4
  • d

    Daniel Nilsen

    04/08/2022, 7:49 AM
    Hi! regarding telemetry. Which of the deployments send telemetry? In the docs it says prefect server. But the server includes apollo, hasura, towel and graphql deployments
    a
    • 2
    • 2
  • t

    Thomas Pedersen

    04/08/2022, 9:01 AM
    Normally our flows run successfully, without any problems. However, we encountered this error for two flow runs:
    Failed to set task state with error: ClientError([{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
        state = self.client.set_task_run_state(
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1598, in set_task_run_state
        result = self.graphql(
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
        raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}]
    In the same timespan, we had 5 flows running, 3 succeeded, only the 2 failed with the above error. The flows run every hour, in the previous hour and next hour everything was running fine.
    start_time                          end_time                            state   state_message
    2022-04-08T02:00:13.555177+00:00	2022-04-08T02:14:19.782245+00:00	Failed	Some reference tasks failed.
    2022-04-08T02:00:13.558573+00:00	2022-04-08T02:14:23.027744+00:00	Success	All reference tasks succeeded.
    2022-04-08T02:00:13.849767+00:00	2022-04-08T02:14:25.492537+00:00	Success	All reference tasks succeeded.
    2022-04-08T02:00:13.929466+00:00	2022-04-08T02:14:23.787185+00:00	Failed	Some reference tasks failed.
    2022-04-08T02:00:14.427653+00:00	2022-04-08T02:21:56.436942+00:00	Success	All reference tasks succeeded.
    a
    • 2
    • 4
  • y

    Yana Burlaka

    04/08/2022, 3:47 PM
    Hello, team Prefect! I wonder if there is a way to use SSO in Prefect Server? And to be more precise I’m interested in Okta as SSO provider. Or this feature is only available in Prefect Cloud Enterprise? Thank you
    k
    • 2
    • 8
  • c

    Christian Nuss

    04/08/2022, 5:32 PM
    heyo! i'm using
    prefect agent docker start
    ... is there a way to prevent the container from being removed once its completed?
    k
    m
    • 3
    • 34
  • d

    Diego Alonso Roque Montoya

    04/09/2022, 2:29 AM
    Is there a way to clear the data of Prefect?
    k
    • 2
    • 7
  • m

    Mariusz Olszewski

    04/11/2022, 5:02 AM
    Is there aby change in security aspects on open source 2.0?
    a
    m
    • 3
    • 4
  • a

    Alex F

    04/11/2022, 4:55 PM
    Hello Prefect Team , I have a question about Shell Task. During PowerShell execution I am doing gzip operation that works then I run this process in shell manually, without any errors. However during Prefect Shell Task execution it's erroring out. with message "archive entry was compressed using an unsupported compression" . I have a feeling Shell Task spinning up empty bash shell and it's not looking into PowerShell libraries installed for same user as the agent is running under during execution. Any idea how to point Shell Task to same user agent Is running under , or how to tell Shell Task path to Powershell dlls ?
    k
    a
    • 3
    • 8
  • n

    Niels Prins

    04/12/2022, 8:27 AM
    Hi, Kind a weird question. I'm upgrading from 0.15.3 to 1.2.0 in our k8s environment. When registering a flow as I was used to do (via
    flow.register('test')
    ) the system gives met a
    405 Client Error: Method Not Allowed
    I'm notecing that when I'm using client.graphql(...) I get the same error, unless I point the client to the graphql endpoint. Now my question(s): 1. Can I get out of the system why it is giving me a 405? 2. Do I need to register a flow at the UI endpoint or at the GRAPHQL endpoint? we used to do the first in 0.15.3 Hope someone can help chrs Niels Nevermind, FIXED it myself. Just for reference for others. You need the graphql endpoint =D
    a
    • 2
    • 1
  • s

    Shuchita Tripathi

    04/12/2022, 3:57 PM
    I have installed prefect orion in a VM. do i need any flag similar to --expose to start the server to see the dashboard? --expose is not working. and in vm, the server has started but I can't access it from the browser. in the security group, i have 4200, 4201, 8080 and few other ports open, as I did with prefect 1.0
    k
    m
    • 3
    • 10
  • s

    Sam Werbalowsky

    04/12/2022, 6:09 PM
    Hi, I have Prefect Deployed via Helm/Kubernetes, version
    0.15.10
    . Will I run in to issues if I am using a later version
    1.2.0
    of Prefect in the image for the agent/dask worker pods?
    k
    • 2
    • 2
  • d

    Daniel Ross

    04/12/2022, 9:31 PM
    Hi Prefect Community! I have encountered an issue where when I run a 'flow of flows' the parent flow fails with the following error:
    Error during execution of task: ValueError('You have not set an API key for authentication.')
    This only happens when I use the DaskExecutor. (Notably the child flows succeed.) However, if I specify a LocalExecutor everything is fine. I assumed that the container where the flow is being executed was not able to reach the server, but I can see the PREFECT__CLOUD__API set properly (to the GraphQL port on the Prefect Server) in the ECS task, and the PREFECT__BACKEND is set to "server". Any ideas on what could be causing this problem?
    a
    • 2
    • 2
  • c

    Christian Nuss

    04/12/2022, 11:19 PM
    hey all! is there a way to get Prefect to Pickle flows including dependencies?
    k
    • 2
    • 9
  • r

    Rohan Paithankar

    04/12/2022, 11:48 PM
    Hi all, how do we access the optimal hardware requirements for Prefect Server? We are expecting a maximum of 10 users at peak time and minimum of 3 users.
    k
    • 2
    • 3
  • b

    Blake

    04/13/2022, 5:37 AM
    This is a bit weird - so I'm using an
    EnvVarSecret()
    and running on Kubernetes, using a Kubernetes Agent. I'm using a customized agent deployement with the following env vars added (populated via Kube secrets):
    - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
              value: 'true'
            - name: AWS_ACCESS_KEY_ID
              valueFrom: 
                secretKeyRef:
                  name: aws-access-key
                  key: aws_access_key
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-secret-key
                  key: aws_secret_key
    The keys end up on the agent no problem - I can see them being passed through in the deployment and pod config. I even started a shell session on the agent pod and I can see the env vars there no problem! However, when I try to reference them in a flow running on that agent, I get errors like this:
    Task 'AWS_SECRET_ACCESS_KEY': Exception encountered during task execution!
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state
        logger=self.logger,
      File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "/usr/local/lib/python3.7/site-packages/prefect/tasks/secrets/env_var.py", line 50, in run
        raise ValueError("Environment variable not set: {}".format(name))
    ValueError: Environment variable not set: AWS_SECRET_ACCESS_KEY
    These flows work locally with the Env Vars in question set, and they work if I set them manually in the "Run" area of the UI's Kubernetes Run Configuration under "Environment Variables". Shouldn't flows running on an agent pick up the env vars from the agent by default?
    d
    a
    • 3
    • 21
  • s

    Shuchita Tripathi

    04/13/2022, 4:06 PM
    Hi, I am considering using prefect 2.0 for my usecase. Since the UI is not yet supported if prefect is installed in VM, I was thinking if someone could help me understand if below points are supported yet, or if there is any plan for it in future release: 1. Prefect has to be installed in a VM in Azure cloud. Will an agent installed somewhere else be able to connect to it? 2. Can I call prefect flows via an API? My requirement is to make a POST call, which will create and then execute flows. These flows will then use Terraform to create resources in Azure. Is this supported in 2.0? 3. I want to store the flows in Azure. In prefect 1.0, this didn't work for me. AWS worked but not Azure. poetry add prefect[azure] gave error and pip install prefect[azure] went on for hours and never ended. I tried multiple versions of pip. Is this supported in 2.0? (This point is the main reason I am considering 2.0) 4. I am running my application in a virtual env (poetry). In prefect 1.0, if I was running the agent inside virtual env, it was always taking localhost, not the IP of server. Outside of virtual env, it was running fine using the IP provided in config.toml. Can I run agent in virtual env in 2.0 using common config.toml? Thank you!
    m
    k
    • 3
    • 8
  • s

    Sharath Chandra

    04/14/2022, 7:06 AM
    Hi I am using prefect to orchestrate my spark jobs. The spark jobs are submitted with
    spark-submit
    using prefect’s
    ShellTask
    . I have created a subclass of ShellTask to invoke the
    spark-submit
    .The spark jobs are running on k8s. There seems to be an issue especially with long running tasks where the spark jobs completes but the prefect task in unaware of the status and continues keep running. I have tried few suggestions including Added to the following configuration to job template
    “<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>”: “false”
    It looks like
    ShellTask
    has some issue in maintaining the status To overcome this I am thinking the following approach: - In map of jobs say
    ["j1", "j2", "j3"]
    , submit all the jobs and not wait for execution completion (In my case since there is no dependency between tasks) - Separate task to monitor the status for all the tasks. This task runs in a loop say for max 1 hour before timeout. This task looks at status for each task and updates the overall status What is the best way to accomplish this looped task on prefect ?
    a
    • 2
    • 6
  • g

    Gaurav kumar

    04/14/2022, 7:26 AM
    How can we pause a task until a specified time and how to resume the same task before the timeout with some external triggers
    a
    • 2
    • 1
  • s

    Slackbot

    04/14/2022, 3:16 PM
    This message was deleted.
    k
    m
    +2
    • 5
    • 31
  • e

    Eric Mauser

    04/14/2022, 5:53 PM
    Is there any way to create an SLA Automation for multiple flows at a time? I have tons of similar flows and want the same SLA on all of them, but can only select one per automation
    k
    • 2
    • 1
  • s

    Sam Werbalowsky

    04/14/2022, 6:04 PM
    is there a best practice regarding which image to use if you are pinning a version for prefect server via helm…I see
    core-1.2.0
    and
    1.2.0
    and then various versions with either python 3.8 or 3.9.
    k
    m
    • 3
    • 13
  • j

    Jovan Sakovic

    04/16/2022, 5:20 PM
    Hi Prefectionists 😛refect-duck: Is it possible to reference an environment variable in the
    config.toml
    ? Specific example, using an external Postgres db, and adding its connection URL in the
    config.toml
    under
    [server.database]
    . It’d be good to keep the URL a secret, but still version control the file. Or perhaps there’s a different way of configuring this, instead of adding it in the
    .toml
    file? Thanks in advance 🙌
    k
    j
    • 3
    • 5
  • s

    Sharath Chandra

    04/18/2022, 6:12 AM
    Hi, I have a master flow for orchestration. I want to trigger one of the downstream flow even if some tasks in the upstream flow has failed. Is this possible to have the https://docs.prefect.io/api/latest/triggers.html#functions at the flow level ?
    a
    k
    • 3
    • 4
  • r

    Ray Tang

    04/18/2022, 3:25 PM
    Hi all, I am trying to follow this medium post to create a dockerized flow, and I am now able to dockerize the flow and upload it to our image repository. https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a But I don't know how to register it to a docker agent? I have a docker agent running using
    prefect agent docker start --show-flow-logs
    I have tried to run
    prefect register --project test-project --path hello_word_dockerized.py
    , but the registration failed because I don't have pandas installed on my docker agent's host (and I am not intended to because I want to have isolated docker images per flows).
    import prefect
    from prefect import task, Flow
    from prefect.run_configs import DockerRun
    from prefect.storage import Local
    import pandas
    
    def get_logger():
        from prefect.utilities.logging import get_logger
        import logging
        import sys
    
        logger = get_logger()
        logger.setLevel('INFO')
        log_stream = logging.StreamHandler(sys.stdout)
        logger.addHandler(log_stream)
    
        return logger
    
    logger = get_logger()
    
    @task(log_stdout=True)
    def hello_world():
        <http://logger.info|logger.info>("Hello, World (Dockerized)")
    
    with Flow("Hello World Dockerized") as flow:
        hello_world()
    
    IMAGE_NAME = "prefect-hello-world-dockerized"
    IMAGE_TAG = "latest"
    
    import os
    print(os.path.basename(__file__))
    flow.storage = Local(path="/opt/prefect/flows/hello_word_dockerized.py", stored_as_script=True)
    
    flow.run_config = DockerRun(image=f"{IMAGE_NAME}:{IMAGE_TAG}", labels=["uat", "docker"])
    k
    a
    • 3
    • 19
  • m

    Marc Lipoff

    04/18/2022, 6:31 PM
    What's the best way to configure a StartFlowRun() task? I'm trying to kick off another flow from the "parent" flow. I'm on cloud, and running v0.15. When I try the following, I get a response that no flow is found:
    StartFlowRun()(
                flow_name="subflow_name",
                project_name="my_project")
    (I tried the query in the cloud api UI, and it does return a flow id) I assume the problem is that my flow is not authenticated to prefect cloud
    a
    • 2
    • 4
  • j

    Jeff LaPorte

    04/18/2022, 9:53 PM
    Is account auth at cloud.prefect.io borked for anyone else?
    ✅ 1
    n
    • 2
    • 5
Powered by Linen
Title
j

Jeff LaPorte

04/18/2022, 9:53 PM
Is account auth at cloud.prefect.io borked for anyone else?
✅ 1
n

nicholas

04/18/2022, 10:04 PM
Hi @Jeff LaPorte - just tried and didn’t have any issues, what issue are you seeing?
j

Jeff LaPorte

04/18/2022, 10:07 PM
Auth appears to succeed and redirect me to cloud.prefect.io with a url containing code and state query params, which are presumably supposed to be passed to the SPA, but I get what looks like a bucket load error:
@nicholas Want to switch to DM and I can give you my account email?
n

nicholas

04/18/2022, 10:10 PM
Yup, shoot me a DM with your account details!
j

Jeff LaPorte

04/18/2022, 10:10 PM
thank you!
View count: 10