nick vazquez
05/27/2021, 1:08 AMlocalhost:4200
although the prefect server/agent were running on a dedicated scheduler box.
Am I missing something for configuring the workers to work properly?
Do they need to point back at the scheduler's ip for logging?
Do I need to run an agent on each machine to pass jobs to the workers?Kevin Kho
05/27/2021, 3:39 AMDaskExecutor
?nick vazquez
05/27/2021, 3:43 AMimport os
import time
import prefect
from prefect import task, Flow, Parameter
from prefect.run_configs import LocalRun
from prefect.executors import LocalDaskExecutor, DaskExecutor
@task
def say_hello(name):
# Add a sleep to simulate some long-running task
time.sleep(10)
# Load the greeting to use from an environment variable
greeting = os.environ.get("GREETING")
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(f"{greeting}, {name}!")
with Flow("hello-flow") as flow:
people = Parameter("people", default=["Arthur", "Ford", "Marvin"])
say_hello.map(people)
import construction
# Configure the `GREETING` environment variable for this flow
flow.run_config = LocalRun(env={"GREETING": "Hello"})
# Use a `LocalDaskExecutor` to run this flow
# This will run tasks in a thread pool, allowing for parallel execution
# flow.executor = LocalDaskExecutor()
flow.executor = DaskExecutor(address="<tcp://192.168.168.114:8786>")
# Register the flow under the "tutorial" project
flow.register(project_name="tutorial", idempotency_key=flow.serialized_hash())
it outputs this in the terminals on the cluster workers
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000002E6E10DAE20>: Failed to establish a new connection: [WinError 10061] No connection
could be made because the target machine actively refused it'))
Kevin Kho
05/27/2021, 3:47 AMnick vazquez
05/27/2021, 3:49 AMKevin Kho
05/27/2021, 3:56 AMnick vazquez
05/27/2021, 3:57 AMKevin Kho
05/27/2021, 4:01 AMnick vazquez
05/27/2021, 4:04 AMZanie
05/27/2021, 2:44 PMnick vazquez
05/27/2021, 11:11 PM