Hi wave I have this existing `prefect yaml` configuration th Prefect Community #ask-community

Hi :wave: I have this existing `prefect.yaml` conf...

Yaron Levi

06/23/2024, 9:12 AM

Hi 👋 I have this existing

prefect.yaml

configuration that works great 👍 and with it deployment are created successfully in Prefect Cloud (We are using ECS push work pools):

Copy code

name: prefect
prefect-version: 2.14.11
build: null
push: null
pull:
  - prefect.deployments.steps.git_clone:
      id: clone-step
      repository: <https://github.com/yuvital/prefect.git>
      credentials: "{{ prefect.blocks.github-credentials.prefect-repo }}"
      branch: main
  - prefect.deployments.steps.pip_install_requirements:
      directory: "{{ clone-step.directory }}"
      requirements_file: requirements.txt
      stream_output: true
deployments:
  - name: migrate_users_to_no_steps_buckets_maof
    entrypoint: jobs/migrate_users_to_no_steps_buckets_maof/migrate_users_to_no_steps_buckets_maof.py:migrate_users_to_no_steps_buckets_maof
    work_pool:
      name: main
  - name: move_users_to_steps_v3_clalit
    entrypoint: jobs/move_users_to_steps_v3_clalit/move_users_to_steps_v3_clalit.py:move_users_to_steps_v3_clalit
    work_pool:
      name: main
  - name: move_users_to_steps_v3_clalit_garmin
    entrypoint: jobs/move_users_to_steps_v3_clalit_garmin/move_users_to_steps_v3_clalit_garmin.py:move_users_to_steps_v3_clalit_garmin
    work_pool:

Yaron Levi

06/23/2024, 9:13 AM

I am trying to re-create the same config but using the python SDK:

Yaron Levi

06/23/2024, 9:13 AM

Copy code

def update_deployment(token, name, path, tags, schedule, parameters, job_variables):    
    repo = GitRepository(url="<https://github.com/yuvital/prefect.git>",branch="main",credentials={"access_token": token})
    entrypoint = f'{path}:{name}'
    print(f'updating deployment for {name} with entrypoint {entrypoint}')
    schedule = CronSchedule(cron=schedule.get('cron'), timezone=schedule.get('timezone')) if schedule else None
    steps = [        
        pip_install_requirements(directory='.', requirements_file='requirements.txt')
    ]
    flow_ref = flow.from_source(source=repo,entrypoint=entrypoint)
    flow_ref.deploy(
        name=name,
        work_pool_name="main-xs",
        job_variables=job_variables,
        tags=tags,
        schedule=schedule,
        parameters=parameters,
        build=False,
        steps=steps)

Yaron Levi

06/23/2024, 9:14 AM

But I am getting related to the “steps” argument and the pip_install_requirements() function:

Yaron Levi

06/23/2024, 9:14 AM

Copy code

File "/home/runner/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/runner/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
TypeError: Flow.deploy() got an unexpected keyword argument 'steps'
sys:1: RuntimeWarning: coroutine 'pip_install_requirements' was never awaited

Yaron Levi

06/23/2024, 9:15 AM

Any idea how to properly define the pip_install_requirements() steps and re-create the same behaviour we had with the prefect.yaml?

6 Views

Open in Slack

Previous Next