https://prefect.io logo
Title
k

Keith Veleba

03/03/2022, 3:15 PM
hello, I'm running flows in AWS ECS and I'm using the "prefecthq/prefect:latest-python3.8" image in my flow's ECSRun run_config. I just added a PostgresFetch task and now im getting an a error that the extras are not installed in the prefect image I'm using. Is there an alternative base image I should be using?TIA
a

Anna Geller

03/03/2022, 3:26 PM
You're right! I was able to reproduce:
@Marvin open "The latest Docker image is missing 'postgres' extra in the image prefecthq/prefect:latest-python3.8"
k

Keith Veleba

03/03/2022, 3:28 PM
@Anna Geller thanks for confirming! I assume you expect the extra to be there then?
a

Anna Geller

03/03/2022, 3:29 PM
I do!
k

Kevin Kho

03/03/2022, 3:33 PM
I think for now you can add it through the
EXTRA_PIP_PACKAGES
env variable in the ECSRun
:upvote: 1
a

Anna Geller

03/03/2022, 3:34 PM
as a workaround you can temporarily use my quick-fix image:
annageller/prefect-postgres:python3.8
πŸ™‚ or deploy your own image in the same way
Kevin's idea is much better and easier!
k

Keith Veleba

03/03/2022, 3:34 PM
i will do that! thanks!
πŸ‘ 1
k

Kevin Kho

03/03/2022, 3:35 PM
Docs is here
k

Keith Veleba

03/03/2022, 6:10 PM
That worked. New issue with that flow, but your suggestion with the extra pip packages worked. In my flow I have:
Error during execution of task: TypeError("get_cloud_url() got an unexpected keyword argument 'as_user'")
happens on this line:
flow_run = create_flow_run(flow_name=DEPENDENT_FLOW_NAME, 
                                project_name=PROJECT_NAME, 
                                parameters={"partner_id": partner}, labels=[ENVIRONMENT, FLOW_CONCURRENCY_LABEL])
Not sure what I'm doing wrong or where to look for answers
k

Kevin Kho

03/03/2022, 6:26 PM
What is your Prefect version? And could you give a longer traceback?
k

Keith Veleba

03/03/2022, 6:28 PM
that's all I see in the cloud ui. Version is 0.15.13
from the logs tab on that flow run
k

Kevin Kho

03/03/2022, 6:30 PM
I think it’s an authentication issue on whatever is kicking off the flow run. Are you using an API key?
k

Keith Veleba

03/03/2022, 6:31 PM
i kicked the flow run off from the UI
k

Kevin Kho

03/03/2022, 6:31 PM
Or it could be a version mismatch between the agent and the Flow
I mean authentication of the agent
k

Keith Veleba

03/03/2022, 6:32 PM
im on 3.8.9 and I think the latest 3.8 container is 3.8.12
major versions shouldnt have an issue right?
k

Kevin Kho

03/03/2022, 6:32 PM
I mean Prefect version not Python version. And I think that should be fine yep
k

Keith Veleba

03/03/2022, 6:34 PM
ecs agent's using the same container. Other flow will run without issue, but this is the only one so far that needs to kick off dependent flows
k

Kevin Kho

03/03/2022, 6:34 PM
What is the container you are using? Base Prefect 1.0?
k

Keith Veleba

03/03/2022, 6:36 PM
prefecthq/prefect:latest-python3.8
with extra_pip_packages prefect[postgres,aws]
k

Kevin Kho

03/03/2022, 6:36 PM
I think that will pull 1.0 which is a mismatch of your registration version 0.15.13 and might be a mismatch of your agent Prefect version
k

Keith Veleba

03/03/2022, 6:37 PM
using the same image for my agent
k

Kevin Kho

03/03/2022, 6:37 PM
Let me dig a bit trying to see what version that line changed
k

Keith Veleba

03/03/2022, 6:37 PM
ok. much appreciated
we're running a prod tenant and a dev tenant in cloud - company name is Segmint if you need to look at our cloud config
k

Kevin Kho

03/03/2022, 6:45 PM
I think I’m good. This is what the line looked like before 1.0 and at 1.0, the
as_user=False
was dropped. I think you might need to re-register in 1.0 to sync everything and I think that should work. What is your storage?
k

Keith Veleba

03/03/2022, 6:45 PM
s3
how do I re-register for 1.0?
my script registers when I run it, it uploads to s3 and I get a response from prefect api
if __name__ == "__main__":
  flow.run_config = ECSRun(image="prefecthq/prefect:latest-python3.8", env={"EXTRA_PIP_PACKAGES": "prefect[postgres,aws]"})   
  flow.register(project_name=PROJECT_NAME, labels=["dev"])
k

Kevin Kho

03/03/2022, 6:47 PM
the registration takes the prefect version of the machine you register on so you need to install 1.0 there before re-registering
k

Keith Veleba

03/03/2022, 6:48 PM
ah
so update my prefect locally
k

Kevin Kho

03/03/2022, 6:50 PM
yes exactly
k

Keith Veleba

03/03/2022, 6:52 PM
guess what? it's Obscure Error Thursday! thanks again to you and @Anna Geller for the top notch support.
πŸ‘ 1
k

Kevin Kho

03/03/2022, 6:53 PM
Of course! πŸ™‚
k

Keith Veleba

03/03/2022, 6:58 PM
that did it - flow seems to be running now
πŸ‘ 1