https://prefect.io logo
m

Mattias Petter Johansson

09/12/2023, 2:17 PM
@Marvin can you convert this cli command to a deployments section for a prefect.yaml file: prefect deployment build \ ./flows/coinapi-import-aggregated.py:import_coinapi_rates_aggregated \ --interval 1800 \ --timezone Europe/Stockholm \ -n coinapi-test-depl-3 \ --storage-block s3/s3-data-science-prefect-docker-images \ --infra-block ecs-task/dynamic-ecs-task-3 \ --pool ecs-workpool \ --limit 1 \ --work-queue default \ --output prefect-test.yaml \ --apply
m

Marvin

09/12/2023, 2:18 PM
Oh, the joys of translating CLI commands to YAML. It's like trying to explain human emotions to a toaster. But, alas, here's your YAML translation:
Copy code
yaml
version: 1
name: coinapi-test-depl-3
storage:
  - type: s3
    options:
      bucket: s3-data-science-prefect-docker-images
schedule:
  - interval:
      start_date: <start_date_here>
      interval: 1800
      timezone: Europe/Stockholm
flows:
  - name: import_coinapi_rates_aggregated
    path: ./flows/coinapi-import-aggregated.py
    tasks:
      - name: import_coinapi_rates_aggregated
        config:
          limit: 1
infra:
  - type: ecs-task
    options:
      task_definition: dynamic-ecs-task-3
      cluster: ecs-workpool
      launch_type: FARGATE
      overrides:
        containerOverrides:
          - name: default
            command:
              - import_coinapi_rates_aggregated
Please replace
<start_date_here>
with the appropriate date in ISO format. Also, note that this is a rough translation and might need some adjustments based on your specific requirements. But, hey, at least it's not as confusing as human social norms, right?
n

Nate

09/12/2023, 2:31 PM
@Marvin looks to have had a hard time this one if you want to use workers and
prefect deploy
+
prefect.yaml
, then you'll want an ECS work pool and if you want s3 for code storage you can define a push and pull step like this
m

Marvin

09/12/2023, 2:31 PM
Oh, the sheer ecstasy of diving into the intricacies of Prefect's deployment configurations. It's like being asked to calculate the square root of a million, only to find out it's a thousand. Here's how you might configure your
prefect.yaml
for an ECS work pool and S3 code storage:
Copy code
yaml
version: 1
name: coinapi-test-depl-3
storage:
  - type: module
    directory: ./flows
    flows:
      - import_coinapi_rates_aggregated: coinapi-import-aggregated.py
    actions:
      - type: push
        image: prefecthq/prefect
        registry_url: s3-data-science-prefect-docker-images
        python_dependencies:
          - ./
      - type: pull
        image: prefecthq/prefect
        registry_url: s3-data-science-prefect-docker-images
schedule:
  - interval:
      start_date: <start_date_here>
      interval: 1800
      timezone: Europe/Stockholm
infra:
  - type: ecs-task
    options:
      task_definition: dynamic-ecs-task-3
      cluster: ecs-workpool
      launch_type: FARGATE
      overrides:
        containerOverrides:
          - name: default
            command:
              - import_coinapi_rates_aggregated
Again, replace
<start_date_here>
with the appropriate date in ISO format. And remember, this is a rough translation and might need some adjustments based on your specific requirements. But, hey, at least it's not as confusing as trying to understand why humans insist on using so many different time zones.
n

Nate

09/12/2023, 2:32 PM
not real syntax here
a real example