@Marvin Can deployment.yaml files execute deployments locally rather than in the cloud or in a docker workpool?

To assist you with your question about executing deployments locally, I'll first check the Prefect version you're interested in, as the syntax and capabilities might differ between Prefect 2.x and 3.x. Could you please confirm or tell me which version you're using or interested in?

@Marvin 2.20.7

In Prefect 2.x, you can execute deployments locally by using the

flow.to_deployment

interface, which allows deployments to be registered and served within the same Python environment. This setup lets you execute and schedule deployments independently, without relying on cloud or Docker environments. You can find more details in the Prefect 2.x tutorial on deployments. However, there are some nuances with local relative imports and other potential issues when executing flows and tasks, as seen in some related GitHub issues. Just be cautious of module imports and ensure your Python environment is set up correctly. Hopefully, this adds a bit of clarity without too much existential dread! If you have any more questions or need further assistance, feel free to ask!

Hi @Marvin Trying to run flows locally that update a dataset from a DWH source... Everything logs locally until flow reaches an

await run deployment

command...

@flow(name="update-dataset", log_prints=True, on_completion=[dataset_update_complete],)
async def update_dataset(logger, dataset, **query_params):

# Local logs fine here...
tokens = update_dataset_metadata(logger, dataset.name, dataset.table_config, dataset.query_config, **query_params)
flow_run_ids = []
tokens_per_job = 5000

for config in dataset.data_deployments:
  for i in range(0, len(tokens), tokens_per_job):
    token_batch = tokens[i : i + tokens_per_job]
    # Here logs disappear locally...
    flow_run = await run_deployment(name=config.deployment_name, parameters={
               "tokens": token_batch, **config.deployment_params}, timeout=config.timeout)
    flow_run_ids.append(flow_run.id)

async with get_client() as client:
  while True:
    await anyio.sleep(5)
    # Here logs disappear locally...
    flow_runs = [await client.read_flow_run(fr_id) for fr_id in flow_run_ids]
    states = [flow_run.state for flow_run in flow_runs]
    if all([state.is_final() for state in states]):
      break

# Events trigger "on_completion" hook, logs aren't seen locally...
return {"dataset_name": dataset.name}

Running Prefect flows locally, especially when using

await run_deployment

, can sometimes lead to logs disappearing due to certain nuances in handling async operations and the environment setup. Here's what I found: 1. Local Execution and Logging: When you execute flows locally with async functions and deployments, it's essential to ensure that your logging setup is correctly capturing logs across different asynchronous contexts. Sometimes, logging might not propagate as expected from asynchronous contexts, especially when dealing with external subprocesses or clients. 2. Managing Logs: Make sure that your logging configuration is appropriate for capturing logs at the desired verbosity. Prefect should log asynchronously, but configurations often need tweaking when running deployments locally to handle logs effectively. You might want to explicitly set up an asynchronous logging handler if you notice missing logs. 3. Use Prefect UI or CLI: Logs can be reviewed from the Prefect UI or CLI if they become cumbersome to catch locally using real-time log streams. However, if you're strictly bound to local logs, exploring detailed logging setups might be necessary. 4. Documentation: The Prefect 2.x tutorial on deployments discusses some basics but doesn't cover logs in-depth. Instead, diving into runtime context guides or similar resources might shed light on better log handling during asynchronous operations. 5. Continued Exploration: There weren't direct GitHub issues found regarding logs disappearing locally with

run_deployment

, suggesting a combined personal setup and Whitepaper solution exploration might be needed. Seems like your logger might have got cold feet somewhere along the way. Give it a comfortable async blanket and it just might stick around to tell the whole story! If this doesn't address your issue entirely, there might be need to further explore specific logging frameworks or Prefect configurations to fully align with your local execution needs. Feel free to ask for more detailed help!