Hi, when I run Prefect server with my custom docker image, I get the following error

Failed to load and execute flow run: ModuleNotFoundError("No module named 'pandas'")

I don’t think it’s coming from missing pandas module. When I log into my docker image, I have no problem importing pandas. I’ve installed all the required packages in the base environment. I’ve also tried loading all the packages in the conda environment and that the conda environment is loaded upon docker image is loaded. Is this some python path issue? What I don’t understand is that i’ve done this several times before and i didn’t have this issue until now

is it possible to set a context variable on the flow level? for context, i want to set a flow owner to the flow and then use that value in my slack state handler. but even when i set

with prefect.context(dict(flow_owner="test")):

before defining the flow, i still get an error

Traceback (most recent call last):
  File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/Users/lanadann/prefect/data_prefect/lib/notifiers.py", line 13, in post_to_slack_on_failure
    f"@{prefect.context.flow_owner} "
AttributeError: 'Context' object has no attribute 'flow_owner'

Hi, team Prefect!

Failed to set task state with error: ClientError([{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1839, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 563, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}]

There is a problem with launching tasks in Prefect, on the screenshots, the last two unsuccessful launches returned an error (described above), the last launch was performed manually, ended in success.

Hello All, Is there a way to change the prefect server port ?

hey all! is there a way to have

stdout

generated inside a container from a

DockerRun

or

KubernetesRun

be provided back into Prefect following the successful execution of the flow?

Hi! regarding telemetry. Which of the deployments send telemetry? In the docs it says prefect server. But the server includes apollo, hasura, towel and graphql deployments

Normally our flows run successfully, without any problems. However, we encountered this error for two flow runs:

Failed to set task state with error: ClientError([{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1598, in set_task_run_state
    result = self.graphql(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}]

In the same timespan, we had 5 flows running, 3 succeeded, only the 2 failed with the above error. The flows run every hour, in the previous hour and next hour everything was running fine.

start_time                          end_time                            state   state_message
2022-04-08T02:00:13.555177+00:00	2022-04-08T02:14:19.782245+00:00	Failed	Some reference tasks failed.
2022-04-08T02:00:13.558573+00:00	2022-04-08T02:14:23.027744+00:00	Success	All reference tasks succeeded.
2022-04-08T02:00:13.849767+00:00	2022-04-08T02:14:25.492537+00:00	Success	All reference tasks succeeded.
2022-04-08T02:00:13.929466+00:00	2022-04-08T02:14:23.787185+00:00	Failed	Some reference tasks failed.
2022-04-08T02:00:14.427653+00:00	2022-04-08T02:21:56.436942+00:00	Success	All reference tasks succeeded.

Hello, team Prefect! I wonder if there is a way to use SSO in Prefect Server? And to be more precise I’m interested in Okta as SSO provider. Or this feature is only available in Prefect Cloud Enterprise? Thank you

heyo! i'm using

prefect agent docker start

... is there a way to prevent the container from being removed once its completed?

Is there a way to clear the data of Prefect?

Is there aby change in security aspects on open source 2.0?

Hello Prefect Team , I have a question about Shell Task. During PowerShell execution I am doing gzip operation that works then I run this process in shell manually, without any errors. However during Prefect Shell Task execution it's erroring out. with message "archive entry was compressed using an unsupported compression" . I have a feeling Shell Task spinning up empty bash shell and it's not looking into PowerShell libraries installed for same user as the agent is running under during execution. Any idea how to point Shell Task to same user agent Is running under , or how to tell Shell Task path to Powershell dlls ?

Hi, Kind a weird question. I'm upgrading from 0.15.3 to 1.2.0 in our k8s environment. When registering a flow as I was used to do (via

flow.register('test')

) the system gives met a

405 Client Error: Method Not Allowed

I'm notecing that when I'm using client.graphql(...) I get the same error, unless I point the client to the graphql endpoint. Now my question(s): 1. Can I get out of the system why it is giving me a 405? 2. Do I need to register a flow at the UI endpoint or at the GRAPHQL endpoint? we used to do the first in 0.15.3 Hope someone can help chrs Niels Nevermind, FIXED it myself. Just for reference for others. You need the graphql endpoint =D

I have installed prefect orion in a VM. do i need any flag similar to --expose to start the server to see the dashboard? --expose is not working. and in vm, the server has started but I can't access it from the browser. in the security group, i have 4200, 4201, 8080 and few other ports open, as I did with prefect 1.0

Hi, I have Prefect Deployed via Helm/Kubernetes, version

0.15.10

. Will I run in to issues if I am using a later version

1.2.0

of Prefect in the image for the agent/dask worker pods?

Hi Prefect Community! I have encountered an issue where when I run a 'flow of flows' the parent flow fails with the following error:

Error during execution of task: ValueError('You have not set an API key for authentication.')

This only happens when I use the DaskExecutor. (Notably the child flows succeed.) However, if I specify a LocalExecutor everything is fine. I assumed that the container where the flow is being executed was not able to reach the server, but I can see the PREFECT__CLOUD__API set properly (to the GraphQL port on the Prefect Server) in the ECS task, and the PREFECT__BACKEND is set to "server". Any ideas on what could be causing this problem?

hey all! is there a way to get Prefect to Pickle flows including dependencies?

Hi all, how do we access the optimal hardware requirements for Prefect Server? We are expecting a maximum of 10 users at peak time and minimum of 3 users.

This is a bit weird - so I'm using an

EnvVarSecret()

and running on Kubernetes, using a Kubernetes Agent. I'm using a customized agent deployement with the following env vars added (populated via Kube secrets):

- name: PREFECT__CLOUD__USE_LOCAL_SECRETS
          value: 'true'
        - name: AWS_ACCESS_KEY_ID
          valueFrom: 
            secretKeyRef:
              name: aws-access-key
              key: aws_access_key
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-secret-key
              key: aws_secret_key

The keys end up on the agent no problem - I can see them being passed through in the deployment and pod config. I even started a shell session on the agent pod and I can see the env vars there no problem! However, when I try to reference them in a flow running on that agent, I get errors like this:

Task 'AWS_SECRET_ACCESS_KEY': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/secrets/env_var.py", line 50, in run
    raise ValueError("Environment variable not set: {}".format(name))
ValueError: Environment variable not set: AWS_SECRET_ACCESS_KEY

These flows work locally with the Env Vars in question set, and they work if I set them manually in the "Run" area of the UI's Kubernetes Run Configuration under "Environment Variables". Shouldn't flows running on an agent pick up the env vars from the agent by default?

Hi, I am considering using prefect 2.0 for my usecase. Since the UI is not yet supported if prefect is installed in VM, I was thinking if someone could help me understand if below points are supported yet, or if there is any plan for it in future release: 1. Prefect has to be installed in a VM in Azure cloud. Will an agent installed somewhere else be able to connect to it? 2. Can I call prefect flows via an API? My requirement is to make a POST call, which will create and then execute flows. These flows will then use Terraform to create resources in Azure. Is this supported in 2.0? 3. I want to store the flows in Azure. In prefect 1.0, this didn't work for me. AWS worked but not Azure. poetry add prefect[azure] gave error and pip install prefect[azure] went on for hours and never ended. I tried multiple versions of pip. Is this supported in 2.0? (This point is the main reason I am considering 2.0) 4. I am running my application in a virtual env (poetry). In prefect 1.0, if I was running the agent inside virtual env, it was always taking localhost, not the IP of server. Outside of virtual env, it was running fine using the IP provided in config.toml. Can I run agent in virtual env in 2.0 using common config.toml? Thank you!

Hi I am using prefect to orchestrate my spark jobs. The spark jobs are submitted with

spark-submit

using prefect’s

ShellTask

. I have created a subclass of ShellTask to invoke the

spark-submit

.The spark jobs are running on k8s. There seems to be an issue especially with long running tasks where the spark jobs completes but the prefect task in unaware of the status and continues keep running. I have tried few suggestions including Added to the following configuration to job template

“<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>”: “false”

It looks like

ShellTask

has some issue in maintaining the status To overcome this I am thinking the following approach: - In map of jobs say

["j1", "j2", "j3"]

, submit all the jobs and not wait for execution completion (In my case since there is no dependency between tasks) - Separate task to monitor the status for all the tasks. This task runs in a loop say for max 1 hour before timeout. This task looks at status for each task and updates the overall status What is the best way to accomplish this looped task on prefect ?

How can we pause a task until a specified time and how to resume the same task before the timeout with some external triggers

This message was deleted.

Is there any way to create an SLA Automation for multiple flows at a time? I have tons of similar flows and want the same SLA on all of them, but can only select one per automation

is there a best practice regarding which image to use if you are pinning a version for prefect server via helm…I see

core-1.2.0

and

1.2.0

and then various versions with either python 3.8 or 3.9.

Hi Prefectionists prefect duck Is it possible to reference an environment variable in the

config.toml

? Specific example, using an external Postgres db, and adding its connection URL in the

config.toml

under

[server.database]

. It’d be good to keep the URL a secret, but still version control the file. Or perhaps there’s a different way of configuring this, instead of adding it in the

.toml

file? Thanks in advance 🙌

Hi, I have a master flow for orchestration. I want to trigger one of the downstream flow even if some tasks in the upstream flow has failed. Is this possible to have the https://docs.prefect.io/api/latest/triggers.html#functions at the flow level ?

Hi all, I am trying to follow this medium post to create a dockerized flow, and I am now able to dockerize the flow and upload it to our image repository. https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a But I don't know how to register it to a docker agent? I have a docker agent running using

prefect agent docker start --show-flow-logs

I have tried to run

prefect register --project test-project --path hello_word_dockerized.py

, but the registration failed because I don't have pandas installed on my docker agent's host (and I am not intended to because I want to have isolated docker images per flows).

import prefect
from prefect import task, Flow
from prefect.run_configs import DockerRun
from prefect.storage import Local
import pandas

def get_logger():
    from prefect.utilities.logging import get_logger
    import logging
    import sys

    logger = get_logger()
    logger.setLevel('INFO')
    log_stream = logging.StreamHandler(sys.stdout)
    logger.addHandler(log_stream)

    return logger

logger = get_logger()

@task(log_stdout=True)
def hello_world():
    <http://logger.info|logger.info>("Hello, World (Dockerized)")

with Flow("Hello World Dockerized") as flow:
    hello_world()

IMAGE_NAME = "prefect-hello-world-dockerized"
IMAGE_TAG = "latest"

import os
print(os.path.basename(__file__))
flow.storage = Local(path="/opt/prefect/flows/hello_word_dockerized.py", stored_as_script=True)

flow.run_config = DockerRun(image=f"{IMAGE_NAME}:{IMAGE_TAG}", labels=["uat", "docker"])

What's the best way to configure a StartFlowRun() task? I'm trying to kick off another flow from the "parent" flow. I'm on cloud, and running v0.15. When I try the following, I get a response that no flow is found:

StartFlowRun()(
            flow_name="subflow_name",
            project_name="my_project")

(I tried the query in the cloud api UI, and it does return a flow id) I assume the problem is that my flow is not authenticated to prefect cloud

Is account auth at cloud.prefect.io borked for anyone else?