Hi folks Does anyone have any luck specifying a schedule in Prefect Community #prefect-community

Hi folks, Does anyone have any luck specifying a s...

Kelvin

07/28/2022, 6:21 PM

Hi folks, Does anyone have any luck specifying a schedule in the deployment in 2.0? Also, how to specify environments? in 2.0 beta, I used to use CronSchedule, and SubprocessFlowRunner with a specific environment, but struggling to do the same in 2.0 GA The documentation seems to be missing some details - in the deployment.yaml, there are "schedule" and "infrastructure -> env" fields, but no specifications on what input is required to define them https://orion-docs.prefect.io/concepts/deployments/#deploymentyaml

✅ 1

Anna Geller

07/28/2022, 6:27 PM

A much easier way is to add a schedule directly from the UI - I personally think this provides a much better UX. Curious - have you tried that?

Kelvin

07/28/2022, 6:42 PM

Ok that works for schedules, thanks Anna! Would you have any pointers on how to use environments also? I used to use SubprocessFlowRunner to run some flows in a separate virtual env

Billy McMonagle

07/28/2022, 6:59 PM

Hi @Anna Geller if I may piggyback here... I strongly prefer to set all configuration in code, including schedules... is it possible to do this at this time?

Khuyen Tran

07/28/2022, 7:35 PM

Yes you can set configurations in the file

deployment.yaml

. You can see what you can specify in the file here

🙏 2

Anna Geller

07/28/2022, 7:39 PM

Kelvin, here is an example for local process and local file system https://github.com/anna-geller/prefect-deployment-patterns/blob/main/a_project_template/deploy_commands_local_fs.bash

Anna Geller

07/28/2022, 7:39 PM

@Billy McMonagle I'll bring this point to the product team

❤️ 1

🙏 1

Billy McMonagle

07/28/2022, 7:57 PM

Appreciate it! The UI is super helpful too and the flexibility is great.

🙏 1

🙌 1

Kelvin

07/29/2022, 1:46 AM

@Khuyen Tran thanks for the link, but I've already attached it in my original post - that link doesn't give the details on what to put in the "schedule" and "env" (environment) fields, to specify schedules and environments respectively @Anna Geller Thanks for the example, but doesn't "--name local" only change the name to "local" for the deployment? I'm looking for something to replicate what we used to do in 2.0b6 beta and before:

Copy code

DeploymentSpec(
    flow_location="myflow.py",
    name="dailyrun",
    schedule=CronSchedule( 
        cron="0 0 * * *",
        timezone="America/New_York"),
    flow_runner=SubprocessFlowRunner(condaenv="specialenv01"),
)

Kelvin

07/29/2022, 2:33 AM

Ok I made some progress, at least for the "using conda env" part I did a bit of "Use the Source, Luke!" and dug into the prefect code I also tried this morning, but seem to be having better luck tonight :-) From what I can see, infrastructure currently supports 3 types https://github.com/PrefectHQ/prefect/blob/20c5f74442e982d5933bfec2f91aa5e14a980f2f/src/prefect/deployments.py

Copy code

infrastructure: Union[DockerContainer, KubernetesJob, Process] = Field(
    default_factory=Process
)

Process is defined at: https://github.com/PrefectHQ/prefect/blob/20c5f74442e982d5933bfec2f91aa5e14a980f2f/src/prefect/infrastructure/process.py#L29 "env" only specifies environment variables so not helpful in my case However, it seems that Process runs the command "python" in the yaml - so if I substitute:

Copy code

command:
  - python
  - -m
  - prefect.engine

with

Copy code

command:
  - /my/path/to/miniconda3/envs/specialenv01/bin/python
  - -m
  - prefect.engine

and deploy a job and run it, it seems to work now

Khuyen Tran

07/29/2022, 5:08 PM

We will include information about how to specify schedule in the

deployment.yaml

file in the docs soon. But for now the easiest way is to use the UI

🙌 1

🙏 1

Matt from DataHouse

07/29/2022, 6:44 PM

second this - docs for the scheduler would be a great add!

5 Views

Open in Slack

Previous Next