Maria
04/05/2022, 2:12 PMFailed to load and execute flow run: ModuleNotFoundError("No module named 'pandas'")
I don’t think it’s coming from missing pandas module. When I log into my docker image, I have no problem importing pandas. I’ve installed all the required packages in the base environment. I’ve also tried loading all the packages in the conda environment and that the conda environment is loaded upon docker image is loaded.
Is this some python path issue? What I don’t understand is that i’ve done this several times before and i didn’t have this issue until nowLana Dann
04/05/2022, 11:02 PMwith prefect.context(dict(flow_owner="test")):
before defining the flow, i still get an error
Traceback (most recent call last):
File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
new_state = self.call_runner_target_handlers(old_state, new_state)
File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
new_state = handler(self.task, old_state, new_state) or new_state
File "/Users/lanadann/prefect/data_prefect/lib/notifiers.py", line 13, in post_to_slack_on_failure
f"@{prefect.context.flow_owner} "
AttributeError: 'Context' object has no attribute 'flow_owner'
Konstantin
04/06/2022, 2:39 PMFailed to set task state with error: ClientError([{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1839, in set_task_run_state
result = self.graphql(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 563, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 7f12b68e-3a3b-45f6-9231-283d30105595: provided a running state but associated flow run 2582c918-369e-4422-ade8-811843f495cf is not in a running state.'}}}]
There is a problem with launching tasks in Prefect, on the screenshots, the last two unsuccessful launches returned an error (described above), the last launch was performed manually, ended in success.Ayah Safeen
04/07/2022, 8:39 AMChristian Nuss
04/07/2022, 4:26 PMstdout
generated inside a container from a DockerRun
or KubernetesRun
be provided back into Prefect following the successful execution of the flow?Daniel Nilsen
04/08/2022, 7:49 AMThomas Pedersen
04/08/2022, 9:01 AMFailed to set task state with error: ClientError([{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1598, in set_task_run_state
result = self.graphql(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'Unable to complete operation. An internal API error occurred.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Unable to complete operation. An internal API error occurred.'}}}]
In the same timespan, we had 5 flows running, 3 succeeded, only the 2 failed with the above error. The flows run every hour, in the previous hour and next hour everything was running fine.
start_time end_time state state_message
2022-04-08T02:00:13.555177+00:00 2022-04-08T02:14:19.782245+00:00 Failed Some reference tasks failed.
2022-04-08T02:00:13.558573+00:00 2022-04-08T02:14:23.027744+00:00 Success All reference tasks succeeded.
2022-04-08T02:00:13.849767+00:00 2022-04-08T02:14:25.492537+00:00 Success All reference tasks succeeded.
2022-04-08T02:00:13.929466+00:00 2022-04-08T02:14:23.787185+00:00 Failed Some reference tasks failed.
2022-04-08T02:00:14.427653+00:00 2022-04-08T02:21:56.436942+00:00 Success All reference tasks succeeded.
Yana Burlaka
04/08/2022, 3:47 PMChristian Nuss
04/08/2022, 5:32 PMprefect agent docker start
... is there a way to prevent the container from being removed once its completed?Diego Alonso Roque Montoya
04/09/2022, 2:29 AMMariusz Olszewski
04/11/2022, 5:02 AMAlex F
04/11/2022, 4:55 PMNiels Prins
04/12/2022, 8:27 AMflow.register('test')
) the system gives met a 405 Client Error: Method Not Allowed
I'm notecing that when I'm using client.graphql(...) I get the same error, unless I point the client to the graphql endpoint.
Now my question(s):
1. Can I get out of the system why it is giving me a 405?
2. Do I need to register a flow at the UI endpoint or at the GRAPHQL endpoint? we used to do the first in 0.15.3
Hope someone can help
chrs
Niels
Nevermind, FIXED it myself. Just for reference for others. You need the graphql endpoint =DShuchita Tripathi
04/12/2022, 3:57 PMSam Werbalowsky
04/12/2022, 6:09 PM0.15.10
. Will I run in to issues if I am using a later version 1.2.0
of Prefect in the image for the agent/dask worker pods?Daniel Ross
04/12/2022, 9:31 PMError during execution of task: ValueError('You have not set an API key for authentication.')
This only happens when I use the DaskExecutor. (Notably the child flows succeed.) However, if I specify a LocalExecutor everything is fine. I assumed that the container where the flow is being executed was not able to reach the server, but I can see the PREFECT__CLOUD__API set properly (to the GraphQL port on the Prefect Server) in the ECS task, and the PREFECT__BACKEND is set to "server".
Any ideas on what could be causing this problem?Christian Nuss
04/12/2022, 11:19 PMRohan Paithankar
04/12/2022, 11:48 PMBlake
04/13/2022, 5:37 AMEnvVarSecret()
and running on Kubernetes, using a Kubernetes Agent.
I'm using a customized agent deployement with the following env vars added (populated via Kube secrets):
- name: PREFECT__CLOUD__USE_LOCAL_SECRETS
value: 'true'
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-access-key
key: aws_access_key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-secret-key
key: aws_secret_key
The keys end up on the agent no problem - I can see them being passed through in the deployment and pod config.
I even started a shell session on the agent pod and I can see the env vars there no problem!
However, when I try to reference them in a flow running on that agent, I get errors like this:
Task 'AWS_SECRET_ACCESS_KEY': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state
logger=self.logger,
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/secrets/env_var.py", line 50, in run
raise ValueError("Environment variable not set: {}".format(name))
ValueError: Environment variable not set: AWS_SECRET_ACCESS_KEY
These flows work locally with the Env Vars in question set, and they work if I set them manually in the "Run" area of the UI's Kubernetes Run Configuration under "Environment Variables".
Shouldn't flows running on an agent pick up the env vars from the agent by default?Shuchita Tripathi
04/13/2022, 4:06 PMSharath Chandra
04/14/2022, 7:06 AMspark-submit
using prefect’s ShellTask
.
I have created a subclass of ShellTask to invoke the spark-submit
.The spark jobs are running on k8s.
There seems to be an issue especially with long running tasks where the spark jobs completes but the prefect task in unaware of the status and continues keep running.
I have tried few suggestions including
Added to the following configuration to job template
“<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>”: “false”
It looks like ShellTask
has some issue in maintaining the status
To overcome this I am thinking the following approach:
- In map of jobs say ["j1", "j2", "j3"]
, submit all the jobs and not wait for execution completion (In my case since there is no dependency between tasks)
- Separate task to monitor the status for all the tasks. This task runs in a loop say for max 1 hour before timeout.
This task looks at status for each task and updates the overall status
What is the best way to accomplish this looped task on prefect ?Gaurav kumar
04/14/2022, 7:26 AMSlackbot
04/14/2022, 3:16 PMEric Mauser
04/14/2022, 5:53 PMSam Werbalowsky
04/14/2022, 6:04 PMcore-1.2.0
and 1.2.0
and then various versions with either python 3.8 or 3.9.Jovan Sakovic
04/16/2022, 5:20 PMconfig.toml
? Specific example, using an external Postgres db, and adding its connection URL in the config.toml
under [server.database]
. It’d be good to keep the URL a secret, but still version control the file.
Or perhaps there’s a different way of configuring this, instead of adding it in the .toml
file?
Thanks in advance 🙌Sharath Chandra
04/18/2022, 6:12 AMRay Tang
04/18/2022, 3:25 PMprefect agent docker start --show-flow-logs
I have tried to run prefect register --project test-project --path hello_word_dockerized.py
, but the registration failed because I don't have pandas installed on my docker agent's host (and I am not intended to because I want to have isolated docker images per flows).
import prefect
from prefect import task, Flow
from prefect.run_configs import DockerRun
from prefect.storage import Local
import pandas
def get_logger():
from prefect.utilities.logging import get_logger
import logging
import sys
logger = get_logger()
logger.setLevel('INFO')
log_stream = logging.StreamHandler(sys.stdout)
logger.addHandler(log_stream)
return logger
logger = get_logger()
@task(log_stdout=True)
def hello_world():
<http://logger.info|logger.info>("Hello, World (Dockerized)")
with Flow("Hello World Dockerized") as flow:
hello_world()
IMAGE_NAME = "prefect-hello-world-dockerized"
IMAGE_TAG = "latest"
import os
print(os.path.basename(__file__))
flow.storage = Local(path="/opt/prefect/flows/hello_word_dockerized.py", stored_as_script=True)
flow.run_config = DockerRun(image=f"{IMAGE_NAME}:{IMAGE_TAG}", labels=["uat", "docker"])
Marc Lipoff
04/18/2022, 6:31 PMStartFlowRun()(
flow_name="subflow_name",
project_name="my_project")
(I tried the query in the cloud api UI, and it does return a flow id)
I assume the problem is that my flow is not authenticated to prefect cloudJeff LaPorte
04/18/2022, 9:53 PMJeff LaPorte
04/18/2022, 9:53 PMnicholas
04/18/2022, 10:04 PMJeff LaPorte
04/18/2022, 10:07 PMnicholas
04/18/2022, 10:10 PMJeff LaPorte
04/18/2022, 10:10 PM