https://prefect.io logo
#prefect-community
Title
# prefect-community
g

Glen Trudeau

09/16/2020, 4:08 PM
Hi Guys. I’ve been working on a prefect server implementation on EC2 using the Fargate agent. I’m running into some errors trying to execute flows using Fargate. The agent would deploy the flow to the Fargate cluster but then would return flow not found. From the logs it looks like it was trying to find the flow inside the Fargate node instead of where it is registered and running on the EC2 instance? I then thought maybe it might be better to try to run this using S3 storage but that is returning the error:
Copy code
Failed to load and execute Flow's environment: TypeError('an integer is required (got type bytes)')
j

josh

09/16/2020, 4:11 PM
Hi @Glen Trudeau this issue commonly arises when the image you are executing your flow in has a different python version compared to the one you serialized your flow with. If you are using S3 storage and the Fargate agent then by default the flow is executed in the
prefecthq/prefect:all_extras
image which uses
Python 3.8
. You can set a different base image to execute your flows in through the environment’s metadata field: https://docs.prefect.io/orchestration/execution/storage_options.html#non-docker-storage-for-containerized-environments
g

Glen Trudeau

09/16/2020, 7:53 PM
I upgraded the prefect server python version to 3.8.3 and registered the flow again. Now instead of giving an error it seems to be in a loop of deploying the flow over and over. I can see the task in Fargate but the logs being generated don’t make much sense
j

josh

09/16/2020, 7:57 PM
I’m not sure what that
Binary
file is, could you post a snippet of what you see?
g

Glen Trudeau

09/16/2020, 8:18 PM
Here is my example flow and fargate agent wrapper as well:
j

josh

09/16/2020, 8:34 PM
FWIW I haven’t seen an output like that before. Would you mind opening an issue on the repo with this information / steps you took to reproduce? Easier to triage there instead of a slack thread 🙂
g

Glen Trudeau

09/16/2020, 8:50 PM
Apologies, I took the wrong output there. This is what I’m seeing in CloudWatch logs:
Copy code
2020-09-16T15:47:58.429-05:00	Flow run 4ed1e234-24ac-40b6-af6e-bb40ddefb671 not found

2020-09-16T15:47:58.431-05:00	Traceback (most recent call last):

2020-09-16T15:47:58.431-05:00	File "/usr/local/bin/prefect", line 8, in <module>

2020-09-16T15:47:58.431-05:00	sys.exit(cli())

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__

2020-09-16T15:47:58.431-05:00	return self.main(*args, **kwargs)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main

2020-09-16T15:47:58.431-05:00	rv = self.invoke(ctx)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke

2020-09-16T15:47:58.431-05:00	return _process_result(sub_ctx.command.invoke(sub_ctx))

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke

2020-09-16T15:47:58.431-05:00	return _process_result(sub_ctx.command.invoke(sub_ctx))

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke

2020-09-16T15:47:58.431-05:00	return ctx.invoke(self.callback, **ctx.params)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke

2020-09-16T15:47:58.431-05:00	return callback(*args, **kwargs)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 33, in flow_run

2020-09-16T15:47:58.431-05:00	return _execute_flow_run()

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 70, in _execute_flow_run

2020-09-16T15:47:58.431-05:00	raise ValueError("Flow run {} not found".format(flow_run_id))

2020-09-16T15:47:58.431-05:00	ValueError: Flow run 4ed1e234-24ac-40b6-af6e-bb40ddefb671 not found
j

josh

09/16/2020, 8:52 PM
Hmm did you happen to restart the server in between runs (without persistence) or delete the run manually from the database? Looks like it can’t find it in the db 🤔
g

Glen Trudeau

09/16/2020, 9:04 PM
The server was up the entire time and the database wasn’t touched so that’s odd
j

josh

09/16/2020, 9:05 PM
Not sure if this would have anything to do with it but on your fargate agent configuration could you also set the env var
PREFECT__BACKEND
to
server
?
It seems like what might be happening is the agent is able to communicate with your server instance however when the Prefect Fargate Task starts up it’s looking at some other instance or something because it can’t find the flow run when querying based on the ID. Really not sure why that would be happening though unless it was attempting to communicate with cloud or something (which in that case it would require an auth token so it seems even less likely)
👍 1
g

Glen Trudeau

09/17/2020, 5:06 PM
That was the issue! Thanks!