Moe
09/12/2024, 1:46 PMfrom prefect import flow
from prefect.runner.storage import GitRepository
from prefect_gitlab import GitLabCredentials
if __name__ == "__main__":
flow.from_source(
source=GitRepository(
url="<https://gitlab.com/.../flows.git>",
branch="test-branch",
credentials=GitLabCredentials.load("git-snowflake-etl")
),
entrypoint="flow_1/ecs_flow_1.py:ecs_flow_1",
).deploy(
name="ecs-flow-1",
work_pool_name="dev-ma-ecs-workpool",
tags=["ECS"]
)
Marvin
09/12/2024, 1:46 PMMoe
09/12/2024, 1:47 PMMarvin
09/12/2024, 1:47 PMGitRepository
with submodules might not work as expected.
- Credentials not being resolved correctly in deployment setups using git_clone
.
Useful GitHub Issues:
- Submodule Clone Issue
- Credential Blocks Issue
Try these steps:
1. Ensure the Git URL is correct and using HTTPS.
2. Verify your GitLabCredentials block is accessible.
3. Maybe try manually injecting credentials as a workaround suggested in the issues.
Good luck!Moe
09/12/2024, 3:21 PMMarvin
09/12/2024, 3:21 PMprefect.yaml
, you can use the overrides
and containerOverrides
attributes. Here’s an example setup based on what I found in the ECS Integration Guide:
yaml
- prefect.deployments.steps.ecs_task_run:
taskDefinition: your-task-def
launchType: FARGATE
cluster: your-cluster
overrides:
containerOverrides:
- name: your-container-name
memory: 2048
cpu: 1024
This configuration lets you customize the ECS task definition parameters directly in the deployment file. Make sure to replace your-task-def
, your-cluster
, and your-container-name
with your actual ECS task definition name, ECS cluster name, and container name respectively.
Check the ECS Worker Documentation for more details.
Good luck!Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by