Kirill Popov11/17/2022, 3:21 PM
jack11/17/2022, 5:58 PM
Fernando Silveira11/17/2022, 6:21 PM
). A couple of notes about this: •
runs every day and to ingests a data partition for that date. The flow is parameterized with the
to ingest. • the flow is organized into a few tasks to ingest data from different clients. So the flow just runs a for loop running
Once ingested this data is useful for a bunch of downstream pipelines. So I'd like to re-use
as a starting point for two other flows, say,
. The way I thought this could be accomplished was by simply running
as a subflow at the beginning of both
. Obviously I don't want to ingest the same data twice. I created a caching key for
parameters. However I quickly stumbled upon the fact that task caches only rely on local storage. I'm running these flows in kubernetes and I was hoping for flows to run concurrently ensuring that ingestion only happens once and the cached status gets re-used in the second flow to run. The fact that task caches are restricted to local storage tells me this is not the way to implement what I wanted. Can someone point me in the right direction here? Happy to discuss more details if that's needed.
Heather DeHaven11/17/2022, 6:41 PM
Kevin Wang11/17/2022, 7:14 PM
Vishy ganesh11/17/2022, 8:53 PM
option but, s3_upload has no basepath.. How do we associate a basepath to the s3_upload option Essentially I don't want the files to land in the bucket root
Jean-Michel Provencher11/17/2022, 9:24 PM
and follow the steps in
git clone <https://github.com/jmprovencher/prefect-tutorial>
Erik Mathis11/17/2022, 9:37 PM
Madison Schott11/17/2022, 10:55 PM
from the function, any ideas?
A 'sync_compatible' method was called from a context that was previously async but is now sync. The sync call must be changed to run in a worker thread to support sending the coroutine for 'load' to the main thread.
Ben Muller11/18/2022, 12:51 AM
to essentially launch 30 concurrent ecs tasks. What I notice is that 5 of the deployment runs start and the other 25 are pending. How do I make this behave similar to prefect 1.0 where I was able to have all of these run at the same time ? Is this a limitation of my agent ? I was under the impression all the agent does is orchestrate the ecs tasks in my aws environment?
Howard11/18/2022, 6:05 AM
Andreas Nigg11/18/2022, 7:27 AM
Tim-Oliver11/18/2022, 11:03 AM
workers with the current main-branch version of Prefect, which works now locally thanks to some very recent changes (thanks a lot 💐). However, I am having troubles getting the logs from tasks which are running on a
. The logs from tasks are written into the slurm-output file, but not propagated back to the flow-log or the cloud UI. Happy, to test some things if it would be helpful.
Joshua Greenhalgh11/18/2022, 12:02 PM
WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")': /
Tushar Gupta11/18/2022, 12:50 PM
Vadym Dytyniak11/18/2022, 1:53 PM
. It doesn't work like we expected. Details in the thread.
Dave11/18/2022, 3:19 PM
within each task and flow. Would this require overriding flow and task initiation, or similar? Thank you in advance for any help!
Miremad Aghili11/18/2022, 5:27 PM
Joshua Grant11/18/2022, 5:30 PM
repo and have tried to use the
to circumvent the issue surrounding
for ECR, but I keep getting the error
. Prefect==2.6.8 & prefect-aws==0.1.8
KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'"
Tibs11/18/2022, 6:05 PM
Vipul11/18/2022, 8:23 PM
Scott Walsh11/18/2022, 9:50 PM
infrastructure setting, but then the sub flow is run by an agent, not the parent flows infrastructure.
Michael Z11/18/2022, 10:30 PM
. This is for prefect 1 btw.
Kimera Joseph11/19/2022, 5:58 AM
komal azram11/19/2022, 6:57 AM
Rob Fowler11/19/2022, 10:04 AM