https://prefect.io logo
Powered by
Title
c

Chris Gunderson

12/09/2022, 4:40 PM
Hi all - Not sure why this error is occurring. I've have included s3fs, gcsfs, and adlfs in the Poetry toml file used to create the Docker image.
Error: 
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 243, in get_filesystem_class
register_implementation(protocol, _import_class(bit["class"]))
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 266, in _import_class
mod = importlib.import_module(mod)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 's3fs'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 389, in filesystem
self._filesystem = fsspec.filesystem(scheme, **self.settings)
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 283, in filesystem
cls = get_filesystem_class(protocol)
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 245, in get_filesystem_class
raise ImportError(bit["err"]) from e
ImportError: Install s3fs to access S3
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 260, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 167, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 463, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 308, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 392, in filesystem
raise RuntimeError(
RuntimeError: File system created with scheme 's3' from base path '<s3://sra-prefect-training>' could not be created. You are likely missing a Python module required to use the given storage protocol.
^^ This error occurs when the ECS Task calls the docker image once the flow is called.
I found this, but it doesn't really help because the infrastructure block is docker, not the ECS Task. https://discourse.prefect.io/t/i-m-getting-an-error-file-system-could-not-be-created-yo[…]d-to-use-the-given-storage-protocol-how-to-solve-that/1459
k

Khuyen Tran

12/09/2022, 7:52 PM
Are you using a Docker block? Have you used specified s3fs in the 
env
field?
🙌 1
c

Chris Gunderson

12/09/2022, 7:56 PM
Hi @Khuyen Tran I am using an ECS Task, which is calling a docker image. The Docker Block is inside Prefect and does have the Env field used as posted above. Unfortunately, I don't think that the ECS Task is referencing the docker container
We are using a cloud formation template to create the ECS Service and task definition
Here is an example of the cloud formation template. I replaced the accountid string with our numeric account number.
spiderrock_dev_ecs_orion_agent_to_prefect_channel.yml
Would I be able to add the EXTRA_PIP_PACKAGES to the environment variable in the container definition?
Like this:
^^Just tried it with the same issue
m

Mason Menges

12/09/2022, 10:22 PM
@Chris Gunderson Just following along I think in this instance you'd need to create an image that contains all of the relevant dependencies and specify that image on the docker container, the Extra pip packages is a simple work around for some use cases but generally speaking hosting an image would be more reliable/preferred in this context. especially with this running through AWS you should be able to push the image to ECR and reference it from there.
c

Chris Gunderson

12/09/2022, 10:27 PM
@Mason Menges I thought the same thing too. I'm not sure why we are getting that error
pyproject.toml
@Mason Menges @Khuyen Tran It looks like Prefect is looking for the venv in /usr/local/lib/python3.9/ instead of /opt/pysetup/.venv Should I copy my venv in both places? Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 243, in get_filesystem_class register_implementation(protocol, _import_class(bit["class"])) File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 266, in _import_class mod = importlib.import_module(mod) File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named 's3fs'
@Ryan Peden Any ideas on the error above?
I'm going to try to revert the docker image back to Prefect 2.6.4 and prefect-aws 0.1.6.
r

Ryan Peden

12/14/2022, 5:32 PM
It looks like the maybe the venv isn't getting used at all? It's just looking at the system python installed packages.
c

Chris Gunderson

12/14/2022, 5:33 PM
Yeah, I was confused by that
r

Ryan Peden

12/14/2022, 5:33 PM
Is the image you are using based on one of Prefect's images? If it's not, then EXTRA_PIP_PACKAGES likely won't work because Prefect images have an entrypoint shell script that installs those.
c

Chris Gunderson

12/14/2022, 5:33 PM
Here is the docker file
Dockerfile
Correct, we cannot use the EXTRA_PIP_PACKAGES, but all the packages are included in the venv
r

Ryan Peden

12/14/2022, 5:36 PM
Makes sense. I think what's happening is that the venv activation needs to run when the container starts. The run in your Dockerfile activates it during the container setup, but I don't think that will carry over to when the container is run.
There are a couple of ways around that I can think of
c

Chris Gunderson

12/14/2022, 5:37 PM
Got it, so put that in the ECS Task command?
r

Ryan Peden

12/14/2022, 5:38 PM
That's one way that I think will work. And if it doesn't, you could add a small shell script that sets the venv and then runs the command that gets passed into the container. Here's the one Prefect uses. The important parts are down near the bottom where the script runs 
exec
on the arguments passed into it
🙌 1
c

Chris Gunderson

12/14/2022, 5:39 PM
Command: - ". /opt/pysetup/.venv/bin/activate && prefect config set PREFECT_API_URL=${PREFECT_API_URL_TEST} && prefect config set PREFECT_API_KEY=${PREFECT_API_KEY_TEST} && prefect agent start -q default"
I think this will work
r

Ryan Peden

12/14/2022, 5:40 PM
It looks like it should 🙂
👍 1
c

Chris Gunderson

12/14/2022, 5:44 PM
Deploying the ECS Service now. Thanks for your help.
r

Ryan Peden

12/14/2022, 6:15 PM
You're welcome. Always happy to help, Chris!
c

Chris Gunderson

12/14/2022, 6:15 PM
It looks like since I didn't specify the ARN in the ECS Task it was using a default image prefecthq/prefect:2.7.1-python3.9
Still working through this
r

Ryan Peden

12/14/2022, 6:16 PM
Is there anything else we can do to assist?
c

Chris Gunderson

12/14/2022, 6:17 PM
Not at the moment. Thanks again
It is still creating a container definition named prefect. Do I need to name my task definition prefect?
r

Ryan Peden

12/14/2022, 7:20 PM
I don't believe that container name is configurable right now as it uses a constant: https://github.com/PrefectHQ/prefect-aws/blob/cdd40bdacec5465050e3fc4f174f4301db294ebc/prefect_aws/ecs.py#L142
7 Views
#prefect-communityJoin Slack
Powered by