Kirill Popov
11/17/2022, 3:21 PMJai P
11/17/2022, 4:18 PMretry_on
functionality is going to be added to Prefect 2 tasks as well? I see it was added in prefect 1.2jack
11/17/2022, 5:58 PMFernando Silveira
11/17/2022, 6:21 PMingestion_flow
). A couple of notes about this:
β’ ingestion_flow
runs every day and to ingests a data partition for that date. The flow is parameterized with the date
to ingest.
β’ the flow is organized into a few tasks to ingest data from different clients. So the flow just runs a for loop running client_ingestion_task(client, date)
for each client
Once ingested this data is useful for a bunch of downstream pipelines. So I'd like to re-use ingestion_flow
as a starting point for two other flows, say, training_flow_1
and training_flow_2
.
The way I thought this could be accomplished was by simply running ingestion_flow
as a subflow at the beginning of both training_flow_{1,2}
. Obviously I don't want to ingest the same data twice. I created a caching key for client_ingestion_task
using the client
and date
parameters. However I quickly stumbled upon the fact that task caches only rely on local storage. I'm running these flows in kubernetes and I was hoping for flows to run concurrently ensuring that ingestion only happens once and the cached status gets re-used in the second flow to run.
The fact that task caches are restricted to local storage tells me this is not the way to implement what I wanted. Can someone point me in the right direction here? Happy to discuss more details if that's needed.Heather DeHaven
11/17/2022, 6:41 PMKevin Wang
11/17/2022, 7:14 PMVishy ganesh
11/17/2022, 8:53 PMbasepath
option but, s3_upload has no basepath..
How do we associate a basepath to the s3_upload option
Essentially I don't want the files to land in the bucket rootJean-Michel Provencher
11/17/2022, 9:24 PMgit clone <https://github.com/jmprovencher/prefect-tutorial>
and follow the steps in README.md
and TUTORIAL.md
Erik Mathis
11/17/2022, 9:37 PMKalise Richmond
11/17/2022, 9:59 PMMadison Schott
11/17/2022, 10:55 PMawait
from the function, any ideas?
A 'sync_compatible' method was called from a context that was previously async but is now sync. The sync call must be changed to run in a worker thread to support sending the coroutine for 'load' to the main thread.
Ben Muller
11/18/2022, 12:51 AMrun_deployment
to essentially launch 30 concurrent ecs tasks. What I notice is that 5 of the deployment runs start and the other 25 are pending.
How do I make this behave similar to prefect 1.0 where I was able to have all of these run at the same time ? Is this a limitation of my agent ? I was under the impression all the agent does is orchestrate the ecs tasks in my aws environment?Howard
11/18/2022, 6:05 AMAndreas Nigg
11/18/2022, 7:27 AMTim-Oliver
11/18/2022, 11:03 AMDaskTaskRunner
workers with the current main-branch version of Prefect, which works now locally thanks to some very recent changes (thanks a lot π). However, I am having troubles getting the logs from tasks which are running on a DaskTaskRunner
which uses dask_jobqueue.SLURMCluster
. The logs from tasks are written into the slurm-output file, but not propagated back to the flow-log or the cloud UI. Happy, to test some things if it would be helpful.Joshua Greenhalgh
11/18/2022, 12:02 PMWARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")': /
Tushar Gupta
11/18/2022, 12:50 PMVadym Dytyniak
11/18/2022, 1:53 PMprefetch-seconds
. It doesn't work like we expected. Details in the thread.Dave
11/18/2022, 3:19 PMget_run_logger().setLevel(level)
within each task and flow. Would this require overriding flow and task initiation, or similar? Thank you in advance for any help!Tushar Gupta
11/18/2022, 5:06 PMMiremad Aghili
11/18/2022, 5:27 PMJoshua Grant
11/18/2022, 5:30 PMdataflow-ops
repo and have tried to use the ECSTask
from prefect-aws
to circumvent the issue surrounding DockerRegistry
for ECR, but I keep getting the error KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'"
. Prefect==2.6.8 & prefect-aws==0.1.8Tibs
11/18/2022, 6:05 PMVipul
11/18/2022, 8:23 PMScott Walsh
11/18/2022, 9:50 PMprocess
infrastructure setting, but then the sub flow is run by an agent, not the parent flows infrastructure.Michael Z
11/18/2022, 10:30 PM*prefect.triggers.all_finished*(_upstream_states_)
. This is for prefect 1 btw.Kimera Joseph
11/19/2022, 5:58 AMKimera Joseph
11/19/2022, 6:27 AMkomal azram
11/19/2022, 6:57 AMRob Fowler
11/19/2022, 10:04 AM