Ross Rochford
08/09/2021, 11:14 AMKevin Kho
08/09/2021, 1:21 PMRoss Rochford
08/09/2021, 1:39 PMwith Flow("list-sum-dask") as flow:
flow.executor = DaskExecutor(address=SCHEDULER_ADDRESS)
Kevin Kho
08/09/2021, 1:39 PMwith Flow("list-sum-dask") as flow:
....
flow.executor = DaskExecutor(address=SCHEDULER_ADDRESS)
Ross Rochford
08/09/2021, 1:43 PMKevin Kho
08/09/2021, 1:47 PMRoss Rochford
08/09/2021, 1:47 PM[2021-08-09 13:42:09+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'list-sum-dask2'
Kevin Kho
08/09/2021, 1:52 PMRoss Rochford
08/09/2021, 1:58 PM[2021-08-09 13:58:54+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'list-sum-dask2'
[2021-08-09 13:58:55,282] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
/usr/local/lib/python3.8/site-packages/distributed/client.py:1190: VersionMismatchWarning: Mismatched versions found
+---------+----------------+---------------+---------------+
| Package | client | scheduler | workers |
+---------+----------------+---------------+---------------+
| blosc | None | 1.9.2 | 1.9.2 |
| lz4 | None | 3.1.3 | 3.1.3 |
| msgpack | 1.0.2 | 1.0.0 | 1.0.0 |
| numpy | None | 1.21.1 | 1.21.1 |
| pandas | None | 1.3.0 | 1.3.0 |
| python | 3.8.11.final.0 | 3.8.0.final.0 | 3.8.0.final.0 |
+---------+----------------+---------------+---------------+
Notes:
- msgpack: Variation is ok, as long as everything is above 0.6
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
[2021-08-09 13:58:55,390] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
[2021-08-09 13:58:55,394] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
DEBUG:agent:Sleeping flow run poller for 2.0 seconds...
[2021-08-09 13:58:57,396] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
[2021-08-09 13:58:57,456] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
Kevin Kho
08/09/2021, 2:01 PMPREFECT___LOGGING___LEVEL="DEBUG"
or you might be able to set the logging level.Ross Rochford
08/09/2021, 2:18 PM2021-08-09 14:13:11+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'list-sum-dask2'
[2021-08-09 14:13:12,755] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
/usr/local/lib/python3.8/site-packages/distributed/client.py:1190: VersionMismatchWarning: Mismatched versions found
+---------+----------------+---------------+---------------+
| Package | client | scheduler | workers |
+---------+----------------+---------------+---------------+
| blosc | None | 1.9.2 | 1.9.2 |
| lz4 | None | 3.1.3 | 3.1.3 |
| msgpack | 1.0.2 | 1.0.0 | 1.0.0 |
| numpy | None | 1.21.1 | 1.21.1 |
| pandas | None | 1.3.0 | 1.3.0 |
| python | 3.8.11.final.0 | 3.8.0.final.0 | 3.8.0.final.0 |
+---------+----------------+---------------+---------------+
Notes:
- msgpack: Variation is ok, as long as everything is above 0.6
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
[2021-08-09 14:13:12,877] DEBUG - agent | No ready flow runs found.
Kevin Kho
08/09/2021, 2:19 PM--show-flow-logs
flag.Ross Rochford
08/09/2021, 2:19 PM9 August 2021,04:09:20 agent INFO Submitted for execution: PID: 158
9 August 2021,04:09:21 prefect.CloudFlowRunner INFO Beginning Flow run for 'list-sum-dask2'
9 August 2021,04:09:21 prefect.CloudFlowRunner DEBUG Using executor type DaskExecutor
9 August 2021,04:09:21 prefect.CloudFlowRunner DEBUG Flow 'list-sum-dask2': Handling state change from Scheduled to Running
Kevin Kho
08/09/2021, 2:23 PMRoss Rochford
08/09/2021, 2:24 PM9 August 2021,04:09:20 agent INFO Submitted for execution: PID: 158
9 August 2021,04:09:21 prefect.CloudFlowRunner INFO Beginning Flow run for 'list-sum-dask2'
9 August 2021,04:09:21 prefect.CloudFlowRunner DEBUG Using executor type DaskExecutor
9 August 2021,04:09:21 prefect.CloudFlowRunner DEBUG Flow 'list-sum-dask2': Handling state change from Scheduled to Running
9 August 2021,04:23:01 prefect-server.Lazarus.FlowRun INFO Rescheduled by a Lazarus process. This is attempt 1.
9 August 2021,04:23:10 agent INFO Submitted for execution: PID: 197
9 August 2021,04:23:11 prefect.CloudFlowRunner INFO Beginning Flow run for 'list-sum-dask2'
9 August 2021,04:23:11 prefect.CloudFlowRunner DEBUG Using executor type DaskExecutor
9 August 2021,04:23:11 prefect.CloudFlowRunner DEBUG Flow 'list-sum-dask2': Handling state change from Scheduled to Running
9 August 2021,04:23:12 prefect.DaskExecutor INFO Connecting to an existing Dask cluster at dask-scheduler:8890
9 August 2021,04:23:15 prefect.CloudFlowRunner DEBUG Checking flow run state...
9 August 2021,04:23:15 prefect.CloudFlowRunner INFO Flow run RUNNING: terminal tasks are incomplete.
import os
import dask
from dask.distributed import Client
import distributed
SCHEDULER_ADDRESS = os.environ['PREFECT__ENGINE__EXECUTOR__DASK__ADDRESS']
def inc(x):
dask.distributed.get_worker().log_event('inc-log', {'x': x})
return x + 1
def square(x):
return x*x
client = Client(SCHEDULER_ADDRESS)
x = client.submit(inc, 10)
print(x.result())
y = client.submit(inc, x)
print(y.result())
L = client.map(inc, range(1000))
list_incremented = client.gather(L)
print(list_incremented)
total = client.submit(sum, L)
print(total.result())
events = client.get_events('inc-log')
print(events)
Kevin Kho
08/09/2021, 2:28 PMRoss Rochford
08/09/2021, 2:28 PMKevin Kho
08/09/2021, 2:30 PMRoss Rochford
08/09/2021, 2:38 PMZanie
08/09/2021, 2:56 PMRoss Rochford
08/09/2021, 3:19 PMZanie
08/10/2021, 11:20 PMRoss Rochford
08/10/2021, 11:31 PMZanie
08/11/2021, 4:40 PM❯ git diff
diff --git a/docker-compose.yml b/docker-compose.yml
index 793dbde..aaddc0a 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -169,11 +169,13 @@ services:
environment:
- DASK_SCHEDULER_ADDRESS=dask-scheduler:8890
- EXTRA_PIP_PACKAGES=prefect
hostname: dask-worker
networks:
- prefect-network
volumes:
- './dask/config:/etc/dask'
+ - './agent/prefect_backend_config.toml:/root/.prefect/backend.toml'
restart: "always"
depends_on:
- dask-scheduler
prefect run --name list-sum-dask --watch --log-level DEBUG
succeeds nowRoss Rochford
08/11/2021, 5:47 PM