https://prefect.io logo
Join Slack
Powered by
hello, I'm running flows in AWS ECS and I'm using...
# prefect-community
k
hello, I'm running flows in AWS ECS and I'm using the "prefecthq/prefect:latest-python3.8" image in my flow's ECSRun run_config. I just added a PostgresFetch task and now im getting an a error that the extras are not installed in the prefect image I'm using. Is there an alternative base image I should be using?TIA
a
You're right! I was able to reproduce:
@Marvin open "The latest Docker image is missing 'postgres' extra in the image prefecthq/prefect:latest-python3.8"
m
k
@Anna Geller thanks for confirming! I assume you expect the extra to be there then?
a
I do!
k
I think for now you can add it through the 
EXTRA_PIP_PACKAGES
env variable in the ECSRun
upvote 1
a
as a workaround you can temporarily use my quick-fix image: 
annageller/prefect-postgres:python3.8
🙂 or deploy your own image in the same way
Kevin's idea is much better and easier!
k
i will do that! thanks!
👍 1
k
Docs is here
k
That worked. New issue with that flow, but your suggestion with the extra pip packages worked. In my flow I have:
Copy code
Error during execution of task: TypeError("get_cloud_url() got an unexpected keyword argument 'as_user'")
happens on this line:
Copy code
flow_run = create_flow_run(flow_name=DEPENDENT_FLOW_NAME, 
                                project_name=PROJECT_NAME, 
                                parameters={"partner_id": partner}, labels=[ENVIRONMENT, FLOW_CONCURRENCY_LABEL])
Not sure what I'm doing wrong or where to look for answers
k
What is your Prefect version? And could you give a longer traceback?
k
that's all I see in the cloud ui. Version is 0.15.13
from the logs tab on that flow run
k
I think it’s an authentication issue on whatever is kicking off the flow run. Are you using an API key?
k
i kicked the flow run off from the UI
k
Or it could be a version mismatch between the agent and the Flow
I mean authentication of the agent
k
im on 3.8.9 and I think the latest 3.8 container is 3.8.12
major versions shouldnt have an issue right?
k
I mean Prefect version not Python version. And I think that should be fine yep
k
ecs agent's using the same container. Other flow will run without issue, but this is the only one so far that needs to kick off dependent flows
k
What is the container you are using? Base Prefect 1.0?
k
prefecthq/prefect:latest-python3.8
with extra_pip_packages prefect[postgres,aws]
k
I think that will pull 1.0 which is a mismatch of your registration version 0.15.13 and might be a mismatch of your agent Prefect version
k
using the same image for my agent
k
Let me dig a bit trying to see what version that line changed
k
ok. much appreciated
we're running a prod tenant and a dev tenant in cloud - company name is Segmint if you need to look at our cloud config
k
I think I’m good. This is what the line looked like before 1.0 and at 1.0, the 
as_user=False
was dropped. I think you might need to re-register in 1.0 to sync everything and I think that should work. What is your storage?
k
s3
how do I re-register for 1.0?
my script registers when I run it, it uploads to s3 and I get a response from prefect api
Copy code
if __name__ == "__main__":
  flow.run_config = ECSRun(image="prefecthq/prefect:latest-python3.8", env={"EXTRA_PIP_PACKAGES": "prefect[postgres,aws]"})   
  flow.register(project_name=PROJECT_NAME, labels=["dev"])
k
the registration takes the prefect version of the machine you register on so you need to install 1.0 there before re-registering
k
ah
so update my prefect locally
k
yes exactly
k
guess what? it's Obscure Error Thursday! thanks again to you and @Anna Geller for the top notch support.
👍 1
k
Of course! 🙂
k
that did it - flow seems to be running now
👍 1
4 Views
Open in Slack
PreviousNext
Powered by