Alexey Stoletny
09/06/2022, 11:16 PMChris Goddard
09/06/2022, 11:45 PMloop.create_task
to try to schedule the sub-flow but I’m getting an unknown error Crash detected! Execution was interrupted by an unexpected exception
- I feel like I’m missing something obvious. (I haven’t done much in asyncio before so it’s very possible)Pierre LIBAULT
09/07/2022, 2:22 AM2022-09-07 11:17:55,962 - distributed.worker - WARNING - Compute Failed
Key: 6b8ff000-ad9a-4de0-ac5a-0b9c35c93fe0
Function: begin_task_run
args: ()
kwargs: {'task': <prefect.tasks.Task object at 0x7ff4aaaff760>, 'task_run': TaskRun(id=UUID('44d28b18-a961-43a8-9f5f-e02134767737'), name='say_goodbye-261e56a8-0', flow_run_id=UUID('16b3db98-966b-44b6-b73b-c4ff4682c878'), task_key='__main__.say_goodbye', dynamic_key='0', cache_key=None, cache_expiration=None, task_version=None, empirical_policy=TaskRunPolicy(max_retries=0, retry_delay_seconds=0.0, retries=0, retry_delay=0), tags=[], state_id=UUID('4543c5be-881c-47e5-9464-4ef76eef5130'), task_inputs={'name': []}, state_type=StateType.PENDING, state_name='Pending', run_count=0, expected_start_time=DateTime(2022, 9, 7, 2, 17, 54, 691980, tzinfo=Timezone('+00:00')), next_scheduled_start_time=None, start_time=None, end_time=None, total_run_time=datetime.timedelta(0), estimated_run_time=datetime.timedelta(0), estimated_start_time_delta=datetime.timedelta(microseconds=56332), state=Pending(message=None, type=PENDING, result=None)), 'parameters': {'name': 'arthur'}, 'wait_for': None, 'result_filesyste':
Exception: "PermissionError(13, 'Permission denied')"
Does everyone know how I can remove this PermissionError ?
I feel that it tries to modify a file or something but I am not sure..Slackbot
09/07/2022, 8:01 AMBarada Sahu
09/07/2022, 8:20 AMPrefectFuture
to a downstream task or does the task result always get evaluated and then passed as an argument?Bal Raj
09/07/2022, 8:23 AMYu Shen
09/07/2022, 9:07 AMwonsun
09/07/2022, 9:38 AMLucien Fregosi
09/07/2022, 10:31 AMGeorgi Yanev
09/07/2022, 10:34 AMAndrew Stewart
09/07/2022, 1:25 PMJavier Ochoa
09/07/2022, 3:05 PM{
flow_group(
where: {
flows: {
project: {
name: { _eq: "sonarlink-dev" }
}
}
}
order_by: {created: desc}
limit: 100
) {
name
id
labels
schedule
flows { id name version project { id name } }
}
}
Josh Paulin
09/07/2022, 3:18 PM@task
decorator and run it like a normal function, it works as expected.
Stepping into the engine code I can see it shows a final_state
of NotReady
here.Soren Daugaard
09/07/2022, 5:01 PMPREFECT_API_KEY
from secrets manager and set it in my Python code when creating a new Prefect client.
Is something like this the right approach:
from prefect.settings import PREFECT_API_KEY
async def start_flow(flow_deployment_name, params):
secret_key = load_my_secret()
with temporary_settings(updates={PREFECT_API_KEY: secret_key}):
client = get_client()
deployment = await client.read_deployment_by_name(name=flow_deployment_name)
run = await client.create_flow_run_from_deployment(deployment.id, parameters=params)
return run.id
I want to avoid using profiles because I would like to avoid having the secret stored on disk.James Phoenix
09/07/2022, 5:48 PMGCS
storage block be used to save data?James Phoenix
09/07/2022, 5:49 PMJames Phoenix
09/07/2022, 5:49 PMfrom prefect import flow, task
from prefect.filesystems import GCS
@flow
async def analyse_youtube_video():
# 1. Download the video from YouTube and upload it to GCS
gcs_block = GCS.load("buffalo-raw-video-files")
# Make some fake JSON data:
data = {"name": "John", "age": 30, "city": "New York"}
result = await gcs_block.write_path(
"video.mp4", data)
print(result)
return 42
James Phoenix
09/07/2022, 5:49 PMsys:1: RuntimeWarning: coroutine 'create_then_begin_flow_run' was never awaited
James Phoenix
09/07/2022, 5:49 PMQwame
09/07/2022, 5:56 PMsqlite3.OperationalError: no such table: json_each
Has anyone experienced this?Arun Giridharan
09/07/2022, 6:10 PMChris Gunderson
09/07/2022, 7:01 PMfrom prefect.blocks.system import *
sraDatabaseSecretName = String.load("sra-database") #This is the name of the secret
Traceback (most recent call last):
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client.py", line 1268, in read_block_document_by_name
response = await self._client.get(
File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1751, in get
return await self.request(
File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client.py", line 279, in send
response.raise_for_status()
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client.py", line 225, in raise_for_status
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url 'http://ephemeral-orion/api/block_types/slug/string/block_documents/name/sra-database?include_secrets=true'
Response: {'detail': 'Block document not found'}
For more information check: https://httpstatuses.com/404Mike Vanbuskirk
09/07/2022, 7:14 PMJoshua Massover
09/07/2022, 7:55 PM[2022-09-07 19:14:16,051] DEBUG - agent | Deleting job prefect-job-c10e0512
• the killing event in my k8s cluster
apiVersion: v1
count: 1
eventTime: null
firstTimestamp: "2022-09-07T19:13:23Z"
involvedObject:
apiVersion: v1
...
kind: Pod
name: prefect-job-c10e0512-9wm4j
...
kind: Event
lastTimestamp: "2022-09-07T19:13:23Z"
message: Stopping container prefect-container-prepare
...
reason: Killing
....
type: Normal
• i can see via metrics that i am not oom'ing or doing anything that seems like it should trigger the job being killed
• a single flow is running on its own node controlled via the kubernetes cluster autoscaler
• i don't see any reason why the cluster autoscaler would be killing this node, and safe-to-evict is set to false.
• my application logs always just end, there's nothing suspicious in the logs
• there aren't obvious patterns to me. it's not the same job, it's not happening after x amount of minutes.
• i've switched to threaded heartbeats, and then most recently turned off heartbeats entirely, and it hasn't fixed it
1. there's a chicken/egg that i'm not sure about. in the agent log, is it issuing a request to the k8s cluster to kill a job? or is it deleted after the kubernetes job kills it for some reason?
2. Any suggestions for how to debug a killed flow in a kubernetes cluster using cluster autoscaling? I can see that it's being killed by the event, it's a herculean task to figure out why it's killed.John Mizerany
09/07/2022, 10:03 PMsys.path.append
to include the module in our PYTHONPATH but that did not work. We are using Git Remote storage but it seems the agent we are using is not able to pick up on the custom files/modules we wrote in the subdirectory (we are still using prefect cloud 1.0 and the UI when we create a run gives us Failed to load and execute flow run: ModuleNotFoundError
)Ankur Sheth
09/07/2022, 10:21 PMEmerson Franks
09/07/2022, 11:40 PMMark Li
09/08/2022, 1:52 AMYoung Ho Shin
09/08/2022, 4:28 AMsqlalchemy
errors when running a test flow with many tasks (>10000) locally. Here's the code I'm running:
https://gist.github.com/yhshin11/1832bc945446a62c5c6152abb9c1a0a5
It seems like the problem has to do with the fact that there are too many tasks that are trying to write to the Orion database at the same time. I tried switching to a Postgres database as described in the [docs](https://docs.prefect.io/concepts/database/), and also adding concurrency limit of 10. Neither seems to fix the issues. Any ideas about how to fix this?
Here's an example of the kind of errors I'm getting:
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/14/3o7r>)