Hi Guys. I’ve been working on a prefect server implementation on EC2 using the Fargate agent. I’m running into some errors trying to execute flows using Fargate. The agent would deploy the flow to the Fargate cluster but then would return flow not found. From the logs it looks like it was trying to find the flow inside the Fargate node instead of where it is registered and running on the EC2 instance? I then thought maybe it might be better to try to run this using S3 storage but that is returning the error:

Failed to load and execute Flow's environment: TypeError('an integer is required (got type bytes)')

Hi @Glen Trudeau this issue commonly arises when the image you are executing your flow in has a different python version compared to the one you serialized your flow with. If you are using S3 storage and the Fargate agent then by default the flow is executed in the

prefecthq/prefect:all_extras

image which uses

Python 3.8

. You can set a different base image to execute your flows in through the environment’s metadata field: https://docs.prefect.io/orchestration/execution/storage_options.html#non-docker-storage-for-containerized-environments

I upgraded the prefect server python version to 3.8.3 and registered the flow again. Now instead of giving an error it seems to be in a loop of deploying the flow over and over. I can see the task in Fargate but the logs being generated don’t make much sense

I’m not sure what that

Binary

file is, could you post a snippet of what you see?

Here is my example flow and fargate agent wrapper as well:

FWIW I haven’t seen an output like that before. Would you mind opening an issue on the repo with this information / steps you took to reproduce? Easier to triage there instead of a slack thread 🙂

Apologies, I took the wrong output there. This is what I’m seeing in CloudWatch logs:

2020-09-16T15:47:58.429-05:00	Flow run 4ed1e234-24ac-40b6-af6e-bb40ddefb671 not found

2020-09-16T15:47:58.431-05:00	Traceback (most recent call last):

2020-09-16T15:47:58.431-05:00	File "/usr/local/bin/prefect", line 8, in <module>

2020-09-16T15:47:58.431-05:00	sys.exit(cli())

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__

2020-09-16T15:47:58.431-05:00	return self.main(*args, **kwargs)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main

2020-09-16T15:47:58.431-05:00	rv = self.invoke(ctx)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke

2020-09-16T15:47:58.431-05:00	return _process_result(sub_ctx.command.invoke(sub_ctx))

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke

2020-09-16T15:47:58.431-05:00	return _process_result(sub_ctx.command.invoke(sub_ctx))

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke

2020-09-16T15:47:58.431-05:00	return ctx.invoke(self.callback, **ctx.params)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke

2020-09-16T15:47:58.431-05:00	return callback(*args, **kwargs)

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 33, in flow_run

2020-09-16T15:47:58.431-05:00	return _execute_flow_run()

2020-09-16T15:47:58.431-05:00	File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 70, in _execute_flow_run

2020-09-16T15:47:58.431-05:00	raise ValueError("Flow run {} not found".format(flow_run_id))

2020-09-16T15:47:58.431-05:00	ValueError: Flow run 4ed1e234-24ac-40b6-af6e-bb40ddefb671 not found

Hmm did you happen to restart the server in between runs (without persistence) or delete the run manually from the database? Looks like it can’t find it in the db 🤔

The server was up the entire time and the database wasn’t touched so that’s odd

Not sure if this would have anything to do with it but on your fargate agent configuration could you also set the env var

PREFECT__BACKEND

to

server

?

That was the issue! Thanks!