Moe
09/12/2024, 1:46 PMfrom prefect import flow
from prefect.runner.storage import GitRepository
from prefect_gitlab import GitLabCredentials
if __name__ == "__main__":
flow.from_source(
source=GitRepository(
url="<https://gitlab.com/.../flows.git>",
branch="test-branch",
credentials=GitLabCredentials.load("git-snowflake-etl")
),
entrypoint="flow_1/ecs_flow_1.py:ecs_flow_1",
).deploy(
name="ecs-flow-1",
work_pool_name="dev-ma-ecs-workpool",
tags=["ECS"]
)
Marvin
09/12/2024, 1:46 PMMoe
09/12/2024, 1:47 PMMarvin
09/12/2024, 1:47 PMGitRepository
with submodules might not work as expected.
- Credentials not being resolved correctly in deployment setups using git_clone
.
Useful GitHub Issues:
- Submodule Clone Issue
- Credential Blocks Issue
Try these steps:
1. Ensure the Git URL is correct and using HTTPS.
2. Verify your GitLabCredentials block is accessible.
3. Maybe try manually injecting credentials as a workaround suggested in the issues.
Good luck!Moe
09/12/2024, 3:21 PMMarvin
09/12/2024, 3:21 PMprefect.yaml
, you can use the overrides
and containerOverrides
attributes. Here’s an example setup based on what I found in the ECS Integration Guide:
yaml
- prefect.deployments.steps.ecs_task_run:
taskDefinition: your-task-def
launchType: FARGATE
cluster: your-cluster
overrides:
containerOverrides:
- name: your-container-name
memory: 2048
cpu: 1024
This configuration lets you customize the ECS task definition parameters directly in the deployment file. Make sure to replace your-task-def
, your-cluster
, and your-container-name
with your actual ECS task definition name, ECS cluster name, and container name respectively.
Check the ECS Worker Documentation for more details.
Good luck!