Jiri Klein
04/14/2023, 3:35 PMprefect deployment inspect flow_name/deployment_name
for my deployment, I can see the parameters in the returned JSON.
Additionally, this means that I can’t override the Deployment parameters for ad-hoc runs of my flows.
Is this an expected behaviour in any of the Prefect Cloud UIs?
question also posted in #prefect-uiNate
04/14/2023, 3:58 PMJiri Klein
04/14/2023, 3:59 PMdeployment.apply(upload=False)
in a CI script from GitHub Actionsclass FlowParameters(BaseModel):
"""Model input parameters."""
host: str
db_name: str
dbt_schema: str
@flow(
name="FLOW NAME",
on_completion=[slack_notifier],
on_failure=[slack_notifier],
)
def flow(flow_parameters: FlowParameters) -> Tuple:
prefect deployment inspect flow_name/deployment_name
I get something like (I’ve REMOVED some values)
{
'id': '29d65ded-0f90-49ff-82b9-eb274b830480',
'created': '2023-04-06T16:01:37.951116+00:00',
'updated': '2023-04-14T15:04:37.882777+00:00',
'name': 'uat',
'version': None,
'description': None,
'flow_id': '67e40ff5-3a08-4dc4-a4ea-1b4ec31cab18',
'schedule': None,
'is_schedule_active': True,
'infra_overrides': {
REMOVED
},
'parameters': {'dbt_schema': 'REMOVED', 'db_name': 'REMOVED', 'host': 'REMOVED'},
'tags': [],
'work_queue_name': 'uat',
'parameter_openapi_schema': {'type': 'object', 'title': 'Parameters', 'properties': {}},
'path': REMOVED,
'entrypoint': REMOVED,
'manifest_path': None,
'storage_document_id': None,
'infrastructure_document_id': '5941200f-8d4d-4a40-82dc-0f7dcd2c9e62',
...
'work_pool_name': 'uat',
'infrastructure': {
'type': 'ecs-task',
...
'cpu': 1024,
'memory': 2048,
...
}
}
Nate
04/14/2023, 4:41 PMfrom prefect import flow
from prefect.deployments import Deployment
from pydantic import BaseModel
class FlowParameters(BaseModel):
"""Model input parameters."""
host: str
db_name: str
dbt_schema: str
@flow(
name="FLOW NAME",
)
def my_flow(params: FlowParameters):
pass
deployment = Deployment.build_from_flow(
flow=my_flow,
apply=True,
name="test-deployment",
)
and if i go to create an ad-hoc run I seeJiri Klein
04/14/2023, 4:47 PMDeployment.build_from_flow
def create_deployment_for_flow(
environment: str,
filename: str,
sub_dir: str,
infrastructure: ECSTask,
overrides: Dict[str, str],
parameters: Optional[Dict[str, Any]] = None,
schedule: Optional[str] = None,
src_part: str = SRC_PART,
) -> Deployment:
"""Create a Deployment object for a flow.
Args:
environment: where we want to deploy to
filename: flow filename
sub_dir: flow sub directory
infrastructure: a loaded prefect 2 infrastructure block
overrides: overrides for the infrastructure block
parameters: optional set of flow parameters
schedule: a cron schedule for the flow run
src_part: the root src where the flows will be stored (see Dockerfile)
Returns:
A Deployment object for the flow
"""
return Deployment(
name=environment,
flow_name=f"{sub_dir}/{filename}".replace("/", "."),
# We assume that a flow will be defined via def flow() in the code
entrypoint=f"{filename}:flow",
path=f"/{src_part}/flows/{sub_dir}",
infrastructure=infrastructure,
work_pool_name=environment,
work_queue_name=environment,
infra_overrides=overrides,
# If `schedule` then use a `CronSchedule`, else use `schedule` as it's `None` anyway
schedule=CronSchedule(cron=schedule, timezone="Etc/UTC")
if schedule
else schedule,
parameters=parameters,
)
Deployment.apply(upload=False)
Nate
04/14/2023, 4:53 PMbuild_from_flow
accepts all the kwargs i.e. allows you to skip_upload
and apply
from there directly if you didn't want two separate steps)
and when you go to the UI, you just see something like this?Jiri Klein
04/14/2023, 5:12 PMValidation of flow parameters failed with error:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 298, in retrieve_flow_then_begin_flow_run
parameters = flow.validate_parameters(flow_run.parameters)
File "/usr/local/lib/python3.9/site-packages/prefect/flows.py", line 344, in validate_parameters
args, kwargs = parameters_to_args_kwargs(self.fn, parameters)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/callables.py", line 63, in parameters_to_args_kwargs
raise SignatureMismatchError.from_bad_params(
prefect.exceptions.SignatureMismatchError: Function expects parameters [] but was provided with parameters ['bar', 'foo']
Nate
04/14/2023, 5:33 PMinspect
?Jiri Klein
04/14/2023, 7:13 PMNate
04/17/2023, 3:46 PMJiri Klein
04/17/2023, 4:50 PMNate
04/17/2023, 4:51 PM