RyanB
11/06/2019, 4:51 PMJoe Schmid
11/06/2019, 4:56 PM@task
def my_task():
import prefect
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Blah blah blah")
RyanB
11/06/2019, 4:59 PMjosh
11/06/2019, 5:07 PMRyanB
11/06/2019, 5:12 PMjosh
11/06/2019, 5:13 PMArnaud Legendre
11/20/2019, 4:33 PMfrom dask.distributed import Client, progress
dask_client = Client()
from prefect import context, Flow, task
from prefect.engine.executors import DaskExecutor
@task
def dummy():
import prefect
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Here")
with Flow("ETL") as flow:
l = dummy()
executor = DaskExecutor(address=dask_client.scheduler_info()['address'])
flow.run(executor=executor)
josh
11/20/2019, 4:58 PMArnaud Legendre
11/21/2019, 8:10 AM[2019-11-21 08:09:45,215] INFO - prefect.FlowRunner | Beginning Flow run for 'ETL'
INFO:prefect.FlowRunner:Beginning Flow run for 'ETL'
[2019-11-21 08:09:45,218] INFO - prefect.FlowRunner | Starting flow run.
INFO:prefect.FlowRunner:Starting flow run.
DEBUG:asyncio:Using selector: SelectSelector
DEBUG:asyncio:Using selector: SelectSelector
[2019-11-21 08:09:46,211] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
INFO:prefect.FlowRunner:Flow run SUCCESS: all reference tasks succeeded
whereas with the default `flow.run()`:
[2019-11-21 08:09:41,849] INFO - prefect.FlowRunner | Beginning Flow run for 'ETL'
INFO:prefect.FlowRunner:Beginning Flow run for 'ETL'
[2019-11-21 08:09:41,853] INFO - prefect.FlowRunner | Starting flow run.
INFO:prefect.FlowRunner:Starting flow run.
[2019-11-21 08:09:41,861] INFO - prefect.TaskRunner | Task 'dummy': Starting task run...
INFO:prefect.TaskRunner:Task 'dummy': Starting task run...
[2019-11-21 08:09:41,863] INFO - prefect.Task: dummy | Here
INFO:prefect.Task: dummy:Here
[2019-11-21 08:09:41,867] INFO - prefect.TaskRunner | Task 'dummy': finished task run for task with final state: 'Success'
INFO:prefect.TaskRunner:Task 'dummy': finished task run for task with final state: 'Success'
[2019-11-21 08:09:41,870] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
INFO:prefect.FlowRunner:Flow run SUCCESS: all reference tasks succeeded
josh
11/21/2019, 2:41 PMArnaud Legendre
11/21/2019, 2:42 PM