Hi I want to automate a simple python process with Prefect b Prefect Community #ask-community

Hi! I want to automate a simple python process wit...

Donny Flynn

10/27/2021, 4:53 PM

Hi! I want to automate a simple python process with Prefect, but I have a laptop I close to take in/out of the office. What would be the easiest online agent to run a scheduled flow? Would a GKE job to poll Prefect cloud as an agent be the best/easiest way? I was exploring Coiled, but I don't think you can set that as an agent, unless I'm missing something...

Anna Geller

10/27/2021, 5:02 PM

Hi @Donny Flynn great question! it’s hard to tell what is the easiest way to set up an agent because it all depends on what infrastructure you use. If you use AWS, I personally think that ECS Fargate is the easiest because it’s serverless, so there is no management overhead on your side to monitor the compute. And here is an example tutorial on how to set it up: https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df But if you use GCP, then GKE is a great idea. Once you created your GKE cluster and have it set as your current kubectl context, then you only need to create an API key in Prefect Cloud UI, and pass it to this command:

Copy code

prefect agent kubernetes install --rbac --key YOUR_API_KEY | kubectl apply --namespace=YOUR_NAMESPACE -f -

Then in your flow, you need to configure KubernetesRun as your run configuration. Here are some examples: https://github.com/anna-geller/packaging-prefect-flows Let us know if you have any issues.

👍 2

Anna Geller

10/27/2021, 5:04 PM

and here are some useful docs: • https://docs.prefect.io/orchestration/agents/kubernetes.html#running-in-cluster • https://docs.prefect.io/orchestration/flow_config/run_configs.html#kubernetesrun • https://docs.prefect.io/orchestration/getting-started/set-up.html#create-an-api-key-and-login

Donny Flynn

10/27/2021, 5:06 PM

Thanks for this @Anna Geller! I will be working on this today, I appreciate the direction

👍 1

Donny Flynn

11/01/2021, 8:02 PM

@Anna Geller, first off, that article was awesome on TDS! Super helpful I believe I have pushed the docker image properly to ecr. I registered the Flow in Prefect and I'm seeing all of my tasks here. But it just says that the flow is scheduled to start and never begun. Any tips on what could have gone wrong? I was getting that error message but I specified the permission in the ECSRun config with the

execution_role_arn

, but now it's not kicking the Flow off

Anna Geller

11/01/2021, 8:05 PM

Thanks a lot! @Donny Flynn I think I may need to update that article: what the error is saying is that when using a custom ECR image, you need to provide the execution role ARN explicitly to the ECSRun. But you correctly noticed that yourself! 🙂 when the flow gets stuck in Scheduled state, it may be because of the label mismatch - which labels did you set on your agents and on your ECSRun? If you followed the article, the label in the demo was “prod”

Donny Flynn

11/01/2021, 8:06 PM

More context towards the bottom when getting to the Docker portion would be great! I spent some time figuring out how to get my

requirements.txt

in properly, but I'm not 100% sure I did that correctly

Donny Flynn

11/01/2021, 8:07 PM

Yes, I have "prod" as the label. It created the task in Fargate, but then nothing happened

Anna Geller

11/01/2021, 8:07 PM

You’re definitely not alone, many users struggle with this - there are some tutorials about that on the roadmap 🙂

👍 1

Donny Flynn

11/01/2021, 8:08 PM

I'm wondering if the import statements at the top just didn't run because I misconfigured the image

Anna Geller

11/01/2021, 8:08 PM

can you share your ECSRun configuration? you can strip the account ID for security

Donny Flynn

11/01/2021, 8:08 PM

Copy code

RUN_CONFIG = ECSRun(
    labels=["prod"],
    task_role_arn="arn:aws:iam::XXXXX:role/prefectTaskRole",
    execution_role_arn="arn:aws:iam::XXXXX:role/prefectTaskRole",
    image="<http://XXXXX.dkr.ecr.us-east-2.amazonaws.com/prefect-custom-image|XXXXX.dkr.ecr.us-east-2.amazonaws.com/prefect-custom-image>",
    run_task_kwargs=dict(cluster="prefectEcsCluster", launchType="FARGATE",),
)

Anna Geller

11/01/2021, 8:09 PM

wait, in your ECSRun the task role and execution role are the same!

Donny Flynn

11/01/2021, 8:10 PM

What's my source of truth for that? On AWS? Or what I pushed?

Donny Flynn

11/01/2021, 8:10 PM

Yes, is that bad? 🙈

Anna Geller

11/01/2021, 8:11 PM

Copy code

RUN_CONFIG = ECSRun(
    labels=["prod"],
    task_role_arn="arn:aws:iam::XXX:role/prefectTaskRole",
    execution_role_arn="arn:aws:iam::XXX:role/prefectECSAgentTaskExecutionRole",
    run_task_kwargs=dict(cluster="prefectEcsCluster", launchType="FARGATE",),
    image="<http://XXX.dkr.ecr.us-east-1.amazonaws.com/community:latest|XXX.dkr.ecr.us-east-1.amazonaws.com/community:latest>"
)

just change the execution role to prefectECSAgentTaskExecutionRole and you should be good to go

Anna Geller

11/01/2021, 8:19 PM

@Donny Flynn What’s my source of truth for that? On AWS? Or what I pushed?

I’d say your source of truth is Dockerfile - you can version-control and modify it anytime. As long as you use one of the official Prefect base images, your custom image should work.

Donny Flynn

11/01/2021, 8:54 PM

I changed the role and it worked. Thank you!

upvote 1

Donny Flynn

11/01/2021, 8:55 PM

As for the Dockerfile, it also worked. I am a little shocked, but I was able to copy a requirements.txt (needed pandas + SQLAlchemy) file to ECR successfully 😆

Anna Geller

11/01/2021, 8:58 PM

nice work! If you need help with Dockerfile in the future, feel free to post it. As for requirements.txt, it’s really 2 lines:

Copy code

COPY requirements.txt .
RUN pip install -r requirements.txt

➕ 1

2 Views

Open in Slack

Previous Next