My continuous integration builds fail when there is a problem in the prefect.yaml file and I would like to be able to write a unit test that validates all the entries so this can be done before the changes are merged and attempted to be deployed. Does anyone validate that entries added to the prefect.yaml file are valid and will not fail to deploy without actually deploying them?

Thanks @Nate, I work with Tom and tried this. I'm running with the "dry run" switch just fine. We usually see two kinds of mistakes in the prefect.yaml file we would want to catch with this flag: 1. The

entrypoint

to a flow function is incorrect, perhaps after a file is moved 2. The

parameters

fail validation, because a string does not match an allowed list of enums, for example Your switch seems to catch case #1 and an exception is raised, but case #2 passes with your dry run flag and fails for an actual deployment, because it looks like this exception happens server-side rather than client-side:

Traceback (most recent call last):
  File "/Users/jduffin/dev/git/eval/fastpass/.venv/lib/python3.10/site-packages/prefect/deployments/runner.py", line 388, in _create
    deployment_id = await client.create_deployment(**create_payload)
  File "/Users/jduffin/dev/git/eval/fastpass/.venv/lib/python3.10/site-packages/prefect/client/orchestration/_deployments/client.py", line 823, in create_deployment
    response = await self.request("POST", "/deployments/", json=payload)
  File "/Users/jduffin/dev/git/eval/fastpass/.venv/lib/python3.10/site-packages/prefect/client/orchestration/base.py", line 53, in request
    return await self._client.send(request)
  File "/Users/jduffin/dev/git/eval/fastpass/.venv/lib/python3.10/site-packages/prefect/client/base.py", line 361, in send
    response.raise_for_status()
  File "/Users/jduffin/dev/git/eval/fastpass/.venv/lib/python3.10/site-packages/prefect/client/base.py", line 162, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url '<http://localhost:4200/api/deployments/>'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'type': 'value_error', 'loc': ['body'], 'msg': "Value error, Validation failed for field 'harvest_dataset_identifier'. Failure reason: 'example_enum_value' is not valid under any of the given schemas", ....

ah yea we could probably add in a line that grabs the parameter schema off the flow and makes sure the provided deployment parameters fit, that'd probably be quite useful thanks for the callout!

Awesome, glad to hear that sounds possible 😁

Thanks Nate, really appreciate your work on this! Your updates seem to be working, though the parameter validation you've added client-side seems to be more strict than what was previously enforced server-side. There are several flows that specify parameters in the prefect.yaml for only a subset of the required flow parameters. Previously this worked fine, and Prefect would set defaults from the prefect.yaml for the provided parameters, and leave the other parameters blank. For example, this worked fine.

@flow
def my_flow(
    arg1: str,
    arg2: str,
    arg3: str
):

prefect.yaml
  - name: 'my_flow deployment'
    description: Description here
    entrypoint: path/to/my_flow.py:my_flow
    path: /opt/prefect/fastpass/
    work_pool: *default_kubernetes_pool
    parameters:
      arg1: 'value1'
      arg2: 'value2'

With your latest changes, this fails parameter validation. Additionally, previously, an extra parameter in the prefect.yaml that is nonexistent in the flow function would be ignored during deployment. It's now failing. Personally I'm okay with this behavior, but others might not expect it. Last thing I noticed is that previously, a single failed deployment along the way would exit the deployment process. We rely on this behavior to fail our CI/CD process when the

prefect deploy

command exits in failure. Now, the

prefect deploy --all

command will always complete and will report on failures at the end, which could affect existing expectations that the command fails if something goes wrong. This is another change that I think we'd be okay with, but might not be expected for others

thanks for trying this out! to be transparent, this is relatively low priority for me right now so I will get to fixes/refinements as possible. feel free to open PRs to my existing branch if you have the capacity or interest. there's some design issues lurking here related to long-standing confusion about where the "source of truth" for deployments lives (i.e. in my prefect.yaml or on the server? how do they sync?) that we should probably engage with before adding a feature like this

Understood, thanks for the time you've already spent on this.