Patrick Tan
12/20/2022, 7:09 PMPatrick Tan
12/20/2022, 7:13 PMAmruth VVKP
12/20/2022, 8:08 PMSanthosh Solomon (Fluffy)
12/20/2022, 9:35 PMAdam Roderick
12/20/2022, 10:41 PM.fetchall()
which isn't usable for large datasets. Any other attempts we've done to use similar logic but using a cursor fails. What is it about the prefect execution environment that prevents us from using snowflake cursors?Samuel Hinton
12/21/2022, 2:04 AMWingmore
12/21/2022, 2:39 AMGows Shaik
12/21/2022, 5:38 AMGows Shaik
12/21/2022, 5:40 AMMalavika S Menon
12/21/2022, 6:06 AMDeepanshu Aggarwal
12/21/2022, 6:35 AMFlow run infrastructure exited with non-zero status code -1.
no extra logs nothing, it also leaves the tasks in running state. any idea why this happens ?YD
12/21/2022, 7:41 AM'state_handlers': [o.p_state_handler]
which sent me an SMS alert on task failure.
how can i achieve this kind of behavior in Prefect 2.0?
thanksYD
12/21/2022, 7:51 AM@task(timeout_seconds=1)
but I get an error: TypeError: task() got an unexpected keyword argument 'timeout_seconds'
What is the proper way to set a timeout time for tasks in Prefect 2 ?Kelvin DeCosta
12/21/2022, 9:31 AMtask
is configured with cache_result_in_memory=False
, can its result only be consumed via state.result()
/ future.result()
?
Calling such tasks directly, i.e.. result = my_task()
, seems to raise the MissingResult
error.Mark NS
12/21/2022, 9:48 AMmyapp
docker container for execution.Aleksandr Liadov
12/21/2022, 11:15 AMJaime Raldua Veuthey
12/21/2022, 11:32 AMAlexandru Anghel
12/21/2022, 1:58 PMCrash detected! Execution was interrupted by an unexpected exception: AttributeError: 'KubeCluster' object has no attribute '__qualname__'
For more context, i'll add some code in the thread.
Thanks!Michał
12/21/2022, 2:24 PMFor more information check: <https://httpstatuses.com/422>
151847.351 | INFO | prefect.agent - Completed submission of flow run '5adc8699-d9a5-4042-85d0-0b072a4873d0'
<frozen runpy>128 RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
151848.970 | INFO | Flow run 'fantastic-tamarin' - Hello from demo-flow!
151848.995 | ERROR | Flow run 'fantastic-tamarin' - Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Client error '422 Unprocessable Entity' for url 'http://prefect-server:4200/api/flow_runs/5adc8699-d9a5-4042-85d0-0b072a4873d0/set_state'Edmund Tian
12/21/2022, 3:03 PMsqlite3.OperationalError
). Anyone know what’s going on?
Context:
I’m using the @flow
and @task
decorators for my Python functions in my app hosted on Google Cloud Run. I’m not using Prefect Cloud. I’m on the following prefect version: prefect==2.3.2
. It’s hard for me to repo because this occurs during my chron job. I haven’t been able to repo locally.Mark NS
12/21/2022, 3:19 PMDavid Steiner Sand
12/21/2022, 6:12 PMJessica Smith
12/21/2022, 7:15 PMPaco Ibañez
12/21/2022, 9:46 PMrun_deployment
. What would be the most efficient way of passing the dataframes around?YD
12/21/2022, 9:50 PMallow_failure
does not work as expected
I am trying to see how can I continue with a workflow after a task failed all the allowed retries.
Sample code:
from prefect import task, flow, get_run_logger, allow_failure
from time import sleep
@task(retries=2, timeout_seconds=3)
def test_1(o, test_retries):
if test_retries:
sleep(5)
else:
raise IOError('Raising some error')
return True
@flow(name='test allow_failure')
def run_test_flow():
result = test_1.submit(True, True)
result = test_1.submit(result, False, wait_for=allow_failure(result))
if __name__ == "__main__":
run_test_flow()
I expect this to run test_1.submit
, fail twice, since the sleep
is larger than the timeout_seconds
, and then run the second test_1.submit
But it does not appear to be working
it does not run the second test_1.submit
@User, @User do you have thoughts on this ?Sam Werbalowsky
12/21/2022, 10:24 PMsubmit
? Even if the flow is written in a way that there is no dependencies between tasks?Gows Shaik
12/22/2022, 4:37 AMJons Cyriac
12/22/2022, 5:03 AMChris Arderne
12/22/2022, 7:56 AMv2
(more details in thread!)
1. 🏗️ Structuring project: We're using v2 with a GCS Storage block, a KubernetesJob block and a Flow Deployment (all three defined in Python files). Because we're using GCS Storage, the flow files are not pip installed
so we can only use relative imports in the flow like from .utils import foo
, which can make local development and bugchecking a bit more error-prone. We will probably shift to just using the Docker images local files, but haven't yet experimented with this in Prefect v2. Are there any best-practises around this? (Project structure in thread)
2. 🏃♀️ Running flows: Prefect v1
had a prefect run ...
command in the CLI, which made it possible to run a local flow. Prefect`v2` doesn't seem to have this? We can't add a if __name__ == "__main__"
to the Flow file, because it has relative imports so running it doesn't work. As a workaround I've added a run.py
in a parent directory that imports and runs it, but this isn't very ergonomic. In either case if I want to pass parameters, I must create my own CLI for doing so... Have I missed something in the CLI, or is there a plan to add this? Or is there a better pattern I should be using instead?James Zhang
12/22/2022, 10:14 AM