Ben Muller
11/02/2022, 8:52 PMBen Muller
11/02/2022, 9:22 PMBradley McLaughlin
11/02/2022, 10:02 PMDavid Prince
11/02/2022, 10:43 PMprefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]"
As far as I understand, this is a flow_run record without a flow. I need some assistance with removing this corrupted metadata as our data pipelines are all majorly delayed.Stephen Lloyd
11/03/2022, 4:01 AMmerlin
11/03/2022, 4:06 AMtmp_path
from pytest
The confusing part was the flow code was found via relative path, but when writing the output to text file on a relative path the output files landed at an ephemeral system temporary folder. Naturally this wouldn't be an issue writing to s3, but for development I like to work in a local environment and write files relative to my project root.
I don't think this is caused by the poetry environment, because the agent is running in the same environment but the distinction in behavior is between deployed flows and flows called from the shell:
# flow.py
import
@task
def write_output(dataframe, filepath):
dataframe.to_csv(filepath)
logger.debug(f"current working directory: {Path().cwd()}")
@flow
def write_dataframe():
write_output(df, 'data/csv/datafile.csv')
# shell invocation
poetry run python src/flow.py
# prefect logs:
| DEBUG | Task run... - current working directory: /Users/merlin/prefect-repo
# deployed flow invocation
poetry shell
prefect deployment run write_dataframe/depl-write
# prefect logs:
| DEBUG | Task run... - current working directory: /private/var/folders/gh/...4r3zq.../T/tmpXookj0s08prefect
My goal here is for my task to write to the same local file path where my project folder is stored.
Is there a standard way to specify the working directory for deployed flow code on a local environment? Maybe this is managed by the config defaults (I couldn't find a likely ENV variable). My next attempt will be to specify an absolute path for the write location, but this is not ideal.iKeepo w
11/03/2022, 4:55 AMTim-Oliver
11/03/2022, 7:24 AM.tif
file? I tried to implement a custom Serializer
which just passes the image through and then wanted to use a custom WritableFileSystem
as storage option that takes care of writing/reading the image as .tif
. So far I did not manage and would be grateful for some advice. I am trying this in Prefect 2.x. In Prefect 1.x I managed to do something like this by implementing a custom LocalResult
.José Duarte
11/03/2022, 10:07 AMprefect.context.get("running_with_backend")
José Duarte
11/03/2022, 10:13 AMContextModel.copy
are broken
https://docs.prefect.io/api-ref/prefect/context/?h=context#prefect.context.ContextModel.copyTim-Oliver
11/03/2022, 11:40 AMJosé Duarte
11/03/2022, 12:39 PMDavid Elliott
11/03/2022, 2:17 PMKubernetesJob
infra?
I’m finding that the agent creates the job + pod just fine (and it the flow + pod run through to completion) but after X seconds (per that timeout parameter) the agent logs Job 'xxxxx': Job did not complete.
per this even though the job is mid-way through running? ie it doesn’t seem to have any negative effect on the flow, it’s just telling me the job didn’t complete even when the job is very much still running..? Feels like something’s not quite right, just wanting to understand what the intention is….Tim-Oliver
11/03/2022, 3:34 PMJai P
11/03/2022, 4:47 PMasync
, etc.)?Javier Ochoa
11/03/2022, 7:29 PMBrian Phillips
11/03/2022, 9:08 PMprefect.tasks.prefect.create_flow_run
are being canceled without any additional info in the logs. Is this expected? Has anyone else encountered similar behavior?Madison Schott
11/03/2022, 9:13 PMschedule_type
signify? Do you only use this if you want the sync to be scheduled outside of a scheduled flow?Madison Schott
11/03/2022, 9:15 PMCole Murray
11/03/2022, 11:36 PMBen Muller
11/04/2022, 2:40 AMget_run_logger()
.
In order to test that function I am using .fn()
but I can not test it because get_run_logger()
failed every time because E prefect.exceptions.MissingContextError: There is no active flow or task run context.
Besides possibly using an optional kwarg in every single task that I have, how can I get around this to test my tasks logic properly?Slackbot
11/04/2022, 3:37 AMIkkyu Choi
11/04/2022, 6:01 AMDeepanshu Aggarwal
11/04/2022, 6:41 AMAttributeError: 'coroutine' object has no attribute 'type'
and the flow run crashes. any idea why this happened.
adding detailed logs in the commentBen Muller
11/04/2022, 8:13 AMMathijs Carlu
11/04/2022, 9:25 AMShruti Hande
11/04/2022, 9:45 AMNic
11/04/2022, 10:55 AMTony Yun
11/04/2022, 11:51 AMVadym Dytyniak
11/04/2022, 1:26 PM