https://prefect.io
Join Slack
Hey all, I’m trying to set up an ECS agent to run code stored in GitHub. Right now, I’m able to run ...
# ask-community
j
Hey all, I’m trying to set up an ECS agent to run code stored in GitHub. Right now, I’m able to run basic code, but all of the parameters set in ECSRun() seem to be ignored, like the docker image to use and the cpu/memory. I’ve seen this in a couple other threads but was unable to follow the solutions (something about overriding?) I was able to choose an image location by setting it in the Run mode in prefect cloud UI, but I had to clear the cpu and memory options or else it would give a “string not valid” error. I’m also wondering if there’s a better way to specify the image location w/out the UI?
k
Hey @Justin Liu, cpu and memory are set on the task, not the container as you have seen but image should be straightforward. How do you specify it right now in the code?
Is it like this?
Copy code
flow.run_config = ECSRun(
    image="example/my-custom-image:latest",
    cpu="2 vcpu",
)
j
yes it is! could you clarify what you mean with the cpu and memory?
also tried using EXTRA_PIP_PACKAGES but it didn’t work out either
k
Did you read this thread yet? I think that is the best one to read.
j
RUN_CONFIG = ECSRun(image='jliubutter/prefect',
yeah i did, so do you have to override the cpu/memory in the ui?
k
We set cpu/memory on tasks, not containers. So even if you see a task definition has Xcpu, but you provide Ycpu in ECSRun, the actual task has Ycpu, You can check this with  
aws ecs describe-tasks
j
oh so its allocating the cpu correctly we just cant see it?
k
Yeah unless you use the 
describe-tasks
there
j
i ran that line and i see the correct values thank you
do you what could be wrong witht he image?
k
That….maybe you can test with the standard Prefect image?
prefecthq/prefect
. Where did you specify EXTRA_PIP_PACKAGES?
j
right, well it always defaults to the prefecthq/prefect, we just cant add extra dependencies if we always use it
RUN_CONFIG = ECSRun(env={"EXTRA_PIP_PACKAGES": "scikit-learn matplotlib"}, cpu=256, memory=512)
k
Oh i’m just asking to test to see if it successfully pulls it down
I don’t think we support the EXTRA_PIP_PACKAGES env variable. That’s a Dask thing right? What Executor are you using?
Did you find the EXTRA_PIP_PACKAGES from our docs?
j
i think we are using local, we didn’t specify
yeah it was on the docs
i can try testing with a different prefecthq/prefect version to see if it changes
k
Could you point me to the page?
j
ill try looking for it now, i think it was under kubernetesrun
ok it just loads prefecthq/prefect:0.14.21 every time unless i specify in the UI
k
Gotcha so that’s 
KubernetesRun
specific and won’t work for 
ECSRun
. I will double check though. Are you using 
DockerStorage
by chance? You might be able to this. Where are you hosting your image?
j
image is on dockerhub at the moment, code pulled from github
k
I am confused why what you have wouldn’t work if you specify a DockerHub image. Is it public (wondering so I can try it)?
j
yeah it is, jliubutter/prefect
also didnt work with other images so i think its a different problem
is the only place we need to specify the image in the ECSRun ?
k
Should be yeah. Is your Flow code simple enough to share?
j
yeah one sec
Copy code
import os
import time

import prefect
from prefect.storage import GitHub
from prefect.run_configs import ECSRun
from prefect import task, Flow, Parameter
from prefect.run_configs import LocalRun

import matplotlib
RUN_CONFIG = ECSRun(image='prefecthq/prefect:X.Y.Z-python3.7', 
                    cpu='2 vcpu', memory='4 GB')
                  
STORAGE = GitHub(repo='user/repo', path='hello.py', ref='main',
                    access_token_secret='prefect-token')
@task
def say_hello():
    logger = prefect.context.get("logger")
    for i in range(200):
      <http://logger.info|logger.info>("Hey")
    

with Flow("hello-world", storage=STORAGE, run_config=RUN_CONFIG) as flow:
    say_hello()
k
Just making sure about this, have you been re-registering the flows?
j
oh could that be it? no i’m just editing the code on github and running
k
Oh yeah you need to re-register these changes so the flow metadata gets registered with Prefect
j
oh you know what that was it! 😅 sorry about that
k
No worries! Happy to help!
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by