Jon
10/10/2022, 2:38 PMAttributeError: 'Parameter' object ...
. Why is the flow trying to run when I register it?Angel Acosta
10/10/2022, 3:24 PMAndrew Reeve
10/12/2022, 8:48 AMOluremi Akinwale
10/12/2022, 4:01 PMZi Yuan
10/17/2022, 7:28 AMBebeto Nyamwamu
10/18/2022, 4:12 AMDaniel Mak
10/19/2022, 8:20 AMdate
into my flow as I would need to use that as a parameter for my pipeline?
E.g
runs on Wed 19Oct 2022 23:00. the date parameter 19Oct2022
is passed a parameter into the flow and will be used in the ETL scriptSam Werbalowsky
10/19/2022, 8:48 PMprefect register
as part of a CI was awesome) so that we don’t have to worry about competing dev environments.
◦ I think maybe what we’d want to do now for this is to use S3 storage, since it looks like git storage is only for public repos right now.
• Different docker images for diferent flows:
◦ It looks like in V2, Agents are what the executors (?) in V1 were. Would using different images for different flows mean that we needed to have different agents available?Brett Naul
10/24/2022, 4:01 PMprefect-gcp
CloudRunJob
? the docs are pretty barebones compared to say the Azure equivalent. for me CloudRunJob.run()
is just hanging infinitely with no output but it's hard to say where I went wrong without an example to go off of.Richard Alexander
10/25/2022, 2:43 PMJon
10/26/2022, 1:03 PMTim-Oliver
10/28/2022, 9:16 AMRabea Yousof
10/30/2022, 10:41 AMterrence
11/01/2022, 3:09 PMeddy davies
11/01/2022, 4:59 PMEthan Homan
11/02/2022, 7:09 PM(DBT job on Snowflake) -> (ML training job).
It would be great to provision a small Fargate task with a Docker image with just the DBT project for the first one, and then provision an ECS task with significantly larger resources and a different Docker image for the ML training job.
Is there a way to do this by creating dependencies between Prefect deployments or by specifying infrastructure at the task or sub-flow level?Michiel Verburg
11/03/2022, 9:09 AMBrad Clark
11/04/2022, 4:14 PMprefect_gcp.cloud_storage
as described here https://prefecthq.github.io/prefect-gcp/. I'm tryin to provide some direction to our data engineersStephen Thibeault
11/07/2022, 2:55 PMKhuyen Tran
11/10/2022, 5:01 PMwait_for
to subflows so that they can wait for upstream tasks. In the example below, subflow
didn’t run because the fails
task failed.
This feature is available in Prefect 2.6.6.Khuyen Tran
11/11/2022, 5:31 PMKhuyen Tran
11/11/2022, 11:05 PMTim-Oliver
11/14/2022, 1:57 PMKhuyen Tran
11/15/2022, 4:59 PMimport asyncio
from prefect import flow, get_run_logger
@flow
async def test_async_flow():
await asyncio.sleep(3)
logger = get_run_logger()
logger.debug("a debug")
<http://logger.info|logger.info>("my info")
logger.warning("some warning")
logger.critical("super critical")
logger.error("error!")
print('all done')
@flow
def test_sync_flow():
test_async_flow()
test_sync_flow()
View more here. Kudos to @Andrew Huang for this cool feature.Khuyen Tran
11/17/2022, 4:11 PMtimeout_seconds=n
and the task takes longer than n
seconds to execute, Prefect will raise a TimeOut error.
View more here. Kudos to @Peyton Runyan for this feature.Jon
11/23/2022, 1:48 AMEdmondo Porcu
11/29/2022, 12:39 AMKhuyen Tran
11/30/2022, 4:32 PMfrom prefect import flow, get_run_logger
@flow
def cow_say():
logger = get_run_logger()
<http://logger.info|logger.info>("I'm in [red]danger[/red]!")
if __name__ == "__main__":
cow_say()
View more styles in Rich’s documentation.John Kang
12/01/2022, 8:40 PMJon
12/09/2022, 7:37 PM