Daniel Manson
10/05/2021, 10:29 PM[download from s3] => [unzip] => [do something interesting] => [upload to somewhere]
as a series of tasks that run sequentially on a single filesystem, but as part of a larger flow where tasks are run on entirely separate filesystems. The advantage of allowing these steps to be full-blown tasks rather than just functions is that you can get dashboard level visibility, retries, and better use of library tasks etc.
A related thought - could you compile a dbt graph into a nested part of a prefect flow, so you get full visibility into what dbt is doing at the level of Prefect. That would be cool.Jacob Blanco
10/05/2021, 11:58 PMGaylord Cherencey
10/06/2021, 1:18 AMbanditelol
10/06/2021, 1:44 AMERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network
when executing prefect server start
.
One solution that's recommended is to add the subnet on docker yaml but I couldn't find a way to do that on prefect. Are there any recommendation on this case (deploy on machine with VPN) or any way to edit the docker-compose.yml
?Adam Brusselback
10/06/2021, 3:24 AMAntti Tupamäki
10/06/2021, 6:27 AMJelle Vegter
10/06/2021, 7:41 AMWill
10/06/2021, 11:01 AMAndreas Tsangarides
10/06/2021, 12:04 PMstored_as_script
argument in Docker() along with path
, but cannot see how I can avoid building and pushing an image for each flow that way...Lucas Giger
10/06/2021, 1:12 PMBob Colner
10/06/2021, 2:25 PMBob Colner
10/06/2021, 3:32 PMAndor TĂłth
10/06/2021, 4:33 PMAndor TĂłth
10/06/2021, 5:05 PMShaoyi Zhang
10/06/2021, 5:38 PMflow.run_config = ECSRun(labels=['prod'],
image='xxxxxxxx',
env={"EXTRA_PIP_PACKAGES": "retry"}
)
I was expecting that although the Kubernetes agent has the label âprodâ, prefect scheduler should still only route flow runs to the ECS agent. However, looks like thatâs not the case. Error:
Flow run ad4bce7b-29cb-4e42-a8c3-eeb9a75f7999 has a `run_config` of type `ECSRun`, only `KubernetesRun` is supported
Kevin Kho
10/06/2021, 7:30 PMAbhas P
10/06/2021, 8:38 PMMatt Alhonte
10/06/2021, 10:17 PMMarwan Sarieddine
10/07/2021, 2:14 AMRyan Sattler
10/07/2021, 3:42 AMMatthias Roels
10/07/2021, 8:51 AMSam Barger
10/07/2021, 12:10 PMZheng Xie
10/07/2021, 2:24 PMAkiva Wolberg
10/07/2021, 2:54 PMconn = psycopg2.connect()
from a task back to the flow, and then use it as an argument in another task?
⢠if so, should I consider separating out the SSH and database connections into separate tasks?
⢠using PrefectSecret appropriately --- I currently have a .env
file and a task to get the environment variables and then use the results as a parameter in my other functions, but I suspect this isn't quite right.Zheng Xie
10/07/2021, 3:53 PMAdam Brusselback
10/07/2021, 6:01 PMVamsi Reddy
10/07/2021, 6:35 PMFailed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n AttributeError("Can\'t get attribute \'RandomNum\' on <module \'prefect.__main__\'
Vamsi Reddy
10/07/2021, 6:35 PMVamsi Reddy
10/07/2021, 6:35 PMBrian Sorahan
10/07/2021, 7:49 PMBrian Sorahan
10/07/2021, 7:49 PMnicholas
10/07/2021, 7:51 PMBrian Sorahan
10/07/2021, 7:51 PMMike Rose
06/15/2022, 2:22 PMnicholas
06/15/2022, 2:25 PMMike Rose
06/15/2022, 2:26 PMnicholas
06/15/2022, 2:30 PM