Michael Urrutia
10/27/2022, 12:51 PMSuresh R
10/27/2022, 1:16 PMDavid Elliott
10/27/2022, 1:22 PMallow_failure()
wrapper for the KubernetesJob
infra - it seems to have no effect when running with SequentialTaskRunner
or ConcurrentTaskRunner
. When I run the same flow locally it works as expected (downstream tasks are ran), but when it’s ran on k8s (with KubernetesJob
infrastructure) it ignores the wrapper and the downstream tasks just go to NotReady
as if the allow_failure()
wrapper weren’t used at all. MRE in 🧵Q
10/27/2022, 1:40 PMPREFECT_API_URL
is not set (PREFECT_ORION_UI_API_URL
is).
printenv | grep -E "PREFECT.*API"
# PREFECT_ORION_UI_API_URL=<https://domain.tld/api>
FlowRunNotifications use PREFECT_API_URL
as base api url and since PREFECT_API_URL
is not set api_url
defaults to a rather useless <http://ephemeral-orion/api>
.
I'm wondering if it would make sense to change this line to use PREFECT_ORION_UI_API_URL
if PREFECT_API_URL
is not set.
I see that PREFECT_ORION_UI_API_URL
defaults to PREFECT_API_URL
, but I think if I set PREFECT_API_URL
to a seemingly external url api calls from ui will stop being routed through loopback and that would be a whole thing.Adam
10/27/2022, 1:49 PMcreate_deployment.py
utility that my CI/CD uses that will create and apply deployments when flows change. Instead of complicated bash, it’s easier to do it in Python. However, I’m unclear how I can get the flow
object from a filepath in this case:
deployment = Deployment.build_from_flow(
flow=my_flow, # how do i get from a filepath to the flow
name="s3-example"
)
In Prefect v1, there was a useful utility called from prefect.utilities.storage import extract_flow_from_file
— my question is does such a utility exist in v2? Or alternatively, is there a better way of doing this?Paco Ibañez
10/27/2022, 1:50 PMJai P
10/27/2022, 2:00 PMDockerContainer
. I'm wondering, does that same logic apply to a KubernetesJob
(since we also have to provide an image for that infra block)?Sean Davis
10/27/2022, 2:15 PMDaniel
10/27/2022, 2:39 PMrun_deployment
. When running the script, I get JSONDecodeError: Expecting value: line 1 column 1 (char 0)
that comes from run_deployment
and is thrown at prefect/client/orion.py", line 1329, in read_deployment_by_name: return schemas.core.Deployment.parse_obj(response.json())
. I edited orion.py
to add print(response.json())
at line 1329 and then that line threw the error, but the json response still prints and it looks like a valid json response object. I'm using prefect 2.6.4. I would appreciate any help in getting the deployment working, thanks!Viet Nguyen
10/27/2022, 2:50 PMTim-Oliver
10/27/2022, 2:59 PMChris Gunderson
10/27/2022, 3:20 PMJehan Abduljabbar
10/27/2022, 4:09 PMPaco Ibañez
10/27/2022, 5:09 PMdammy arinde
10/27/2022, 5:28 PMJon Young
10/27/2022, 6:49 PMTypeError: not all arguments converted during string formatting
is there a way to get the parameter's value without giving function with a task decorator?Carlo
10/27/2022, 7:15 PMAndreas Nigg
10/27/2022, 7:54 PMJens Freund
10/27/2022, 11:01 PMFile "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:16:29.384 | INFO | prefect.infrastructure.process - Process 'upbeat-crab' exited cleanly.
To solve the PermissionError, I tried two approaches, which unfortunately both failed:
1. Try to change the PREFECT_LOCAL_STORAGE_PATH
2. Running Powershell, from which I start Prefect, as admin
My problem seems to be practically the same as that reported by @Mike He in the discourse forum (https://discourse.prefect.io/t/agent-debugging-agent-execution-error-with-ftp-storage-in-windows/1587) and here in Slack (https://prefect-community.slack.com/archives/C03D12VV4NN/p1663319368100279). (The only difference I see is that in my case, the last error of the traceback comes from the paramiko library instead of fsspec.) I saw that he already developed a workaround, which however includes changing the Prefect source code which I have not tried yet.
I would write details, e.g. the full traceback, in the thread. If anyone had an idea or a hint how this problem could be solved, preferably without manually changing the Prefect source code, you would help me again a lot. Thank you very much in advance!Andrei Tulbure
10/28/2022, 8:25 AMNic
10/28/2022, 9:21 AMdef cache_within_flow_run(context, parameters):
return f"{context.flow_run_id}-{task_input_hash(context, parameters)}"
How would i pass the flow_run_id into the task this function and further into the task decorator?
@task(cache_key_fn=cache_within_flow_run)
def cached_task():
print('running an expensive operation')
return 42
Camilo Fernandez
10/28/2022, 9:49 AMDeepanshu Aggarwal
10/28/2022, 11:48 AMCarlo
10/28/2022, 1:02 PM23:02:20.896 | INFO | prefect.engine - Engine execution of flow run '682567a2-e507-4aa1-b5c7-25bb2cc2573f' aborted by orchestrator: This run has already terminated.
Nic
10/28/2022, 1:05 PMJon Young
10/28/2022, 2:11 PMPaco Ibañez
10/28/2022, 2:23 PMvholmer
10/28/2022, 2:44 PMawait my_task.submit(_flow_name_=FLOW_NAME, _env_=env, _run_name_=run_name, _wait_for_=[preprocess], _parameters_=parameters)
However after upgrading to 2.6.5 my_task isn't even being run. My workaround is to just run this:
await my_task(_flow_name_=FLOW_NAME, _env_=env, _run_name_=run_name, _wait_for_=[preprocess], _parameters_=parameters)
But it seems really weird that submit should have stopped working. Has anyone else ran into this issue?Arun Giridharan
10/28/2022, 3:36 PMstartFlowRun
method. Initially when I was calling the first flow without using the wait
parameter, the flows worked fine, but after I added the parameter, the parent flow failed with TypeError: Object of type SUCCESS is not JSON serializable
. I was wondering if this is a known issue. Prefect version is 1.4.0Mansour Zayer
10/28/2022, 4:01 PMorder by column_name nulls last
? I have this line order_by: {start_time: desc}
in my query.
More context in commentMansour Zayer
10/28/2022, 4:01 PMorder by column_name nulls last
? I have this line order_by: {start_time: desc}
in my query.
More context in commentnull
(maybe some bug in prefect 1 cloud), and in my request, I'm ordering by start_date (order_by: {start_time: desc}
), and I always get this null as the first one, which is breaking my code. I want null to be last, not first.Kalise Richmond
10/28/2022, 9:24 PM