Running into an issue referencing Anna's dataflow-...
# prefect-community
c
Running into an issue referencing Anna's dataflow-ops template on deploying the ECS agent.
Copy code
"containerDefinitions": [
      "entryPoint": [
        "bash",
        "-c"
      ],
      "command": [
        "prefect agent start -q ecs-worker"
      ],
results in error:
Usage: prefect agent [OPTIONS] COMMAND [ARGS]...
Try 'prefect agent -h' for help.
Error: No such command 'start'.
Copy code
"containerDefinitions": [
      "entryPoint": [
        "bash",
        "-c"
      ],
      "command": [
        "prefect","agent","start","-q","ecs-worker"
      ],
results in what appears to be calling "prefect" with nothing else, as the logs just keep repeating the menu stdout:
t
Try:
Copy code
"entryPoint": [],
            "command": [
                "/bin/bash",
                "-c",
                "prefect agent start --work-queue 'ecs-worker'"
            ],
b
I’m not a prefect person, but this definition for my agent works on my config:
Copy code
- Name: prefect-agent
          Image: !Ref PrefectImage
          Environment:
            - Name: PREFECT_API_URL
              Value: <http://127.0.0.1:4200/api>
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref OrionLogGroup
              awslogs-stream-prefix: agent
          Command: [prefect, agent, start, -q, default]
c
great suggestions!! @Tim Helfensdörfer giving this a go now
@badasstronaut just curious, what's your entrypoint here?
b
@chicago-joe Entrypoint is not a required property for container definitions: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html#ECS-Type-ContainerDefinition-entryPoint I used a base image from Prefect, added a few dependency layers (e.g. pip installs, apt installs), then pushed it to ECR. So the entrypoint is whatever entrypoint is defined by the base Prefect image.
a
Did you use the cloud formation template? it's much easier than manually deploying task definitions https://github.com/anna-geller/dataflow-ops/blob/main/infrastructure/ecs_cluster_prefect_agent.yml
🙌 1
c
@Anna Geller I had followed the ecs_agent setup instructions within the prefect-recipes repo a few weeks ago --- I will redo and follow the cloud formation template. Moving forward, should I rely on this as a base configuration template? Not a terrible thing to have so many resources, but I'm starting to lose track of which one is the recommended way to go hahaha
that worked!! I'm running into another issue right now, and this is going to be a very very dumb question so please forgive me lol.
KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'."
@Anna Geller in your deploy.sh script, you call python blocks/docker.py but I don't see this in the repo, only ecs_task.py. Am I missing something?
🙏 1
💯 1
a
your spot on, my mistake, corrected
this repo right now assumes that you are deploying everything with this manually triggered GitHub Actions workflow rather than manually, so the error you see wouldn't exist if you would trigger it as shown in the video demo and blog post https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/ecs_prefect_agent.yml https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc

https://youtu.be/Eemq2X9XrlE

@chicago-joe thanks a lot for feedback, keep it coming, I adjusted some things in the repo based on your feedback. Generally use recipes that are helpful to you, I'm focused more on quantity to cover a large breadth of use cases, but for quality, you may check the official prefect-recipes
❤️ 1