Hi. At last I'm properly getting to 2.0! :success-...
# prefect-community
Hi. At last I'm properly getting to 2.0! success kid Seems I'm running into the same issue as Keith here Getting
ModuleNotFoundError: No module named 'gcsfs'
It's strange because the environment that my agent is running in has
installed and I have
in my Docker Block... Any tips?
I've created deployments from a Python objects. I'm using a GCS Storage Block and Docker container infra block. I created the Docker Container block via the Cloud UI and used only defaults (i.e. left fields empty), except for Env=
Copy code
  "EXTRA_PIP_PACKAGES": "gcsfs",
  "GOOGLE_APPLICATION_CREDENTIALS": "/opt/prefect/dev-user-staljaard.json"
Volumes =
[ "/home/stephan/di/dev-user-staljaard.json:/opt/prefect/dev-user-staljaard.json" ]
Auto Remove = true
As an alternative, you can build a custom image, this might help -- I believe this could be an issue with the volume attachment. You can create a GitHub issue as well if this doesn't work (because it should) and you can tag me there, I'd be happy to reproduce both options (volumes and custom image)
Eventually I'm going to use a custom image anyway, but for testing things out I wanted to keep it simple 😅 👍 I'll try to get to it tonight
👍 2
blob attention gif 1
🙏 2
I couldn't get to it on Friday, but am busy drafting the issue. While doing so, testing again, I now get a lot of
Copy code
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/xxx/workspaces/xxx/deployments/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
Any issues currently with the API for which I should wait before testing again and completing the ticket? https://prefect.status.io/ seems clear.
I managed to get around this by manually deleting all flows & deployments Even in the Cloud GUI I got errors preventing deletion. I had to Ctrl + F5, after which I was able to delete all. I'll get back to the Docker problems.
Hi Stephan! How are things going on this? I can try to reproduce now - did you open an issue?
Prefect API is fine, just ran some flows - all good on that end
👍 1
I ran into other issues as well (flow runs failing when running using a Docker Container block - complaining about
AttributeError: module 'docker' has no attribute 'from_env'
I'm trying out a Subprocess block for now; need to make progress. I'll revisit gain and create a detailed issue.
👍 1
so I just did the following and it worked:
Copy code
import json
from prefect.infrastructure import DockerContainer
from prefect.filesystems import GCS

service_account_info = json.load(open("/Users/anna/sa.json"))
gcs = GCS(
gcs.save("dev", overwrite=True)

docker_block = DockerContainer(
    image="prefecthq/prefect:2-python3.10", env={"EXTRA_PIP_PACKAGES": "gcsfs"}
docker_block.save("dev", overwrite=True)

pip install gcsfs
prefect deployment build -n dev -q dev -sb gcs/dev -ib docker-container/dev -a flows/healthcheck.py:healthcheck

prefect deployment run healthcheck/dev
prefect agent start -q dev
will now change to 2.4.0 image
googling your error, SO says: "I had pip installed docker-py, but the current version of the package is simply called "docker" on pypi."
could be package version mismatch - I'd recommend creating a new conda environment and installing latest Prefect version there, then starting the agent -- It should work then, but if it doesn't please create an issue and link here. Thanks Stephan!
I'll investigate again and create an issue, and give feedback 👍
wait, can it be that your flow file is called docker.py?
🙅 1
if so, please change it to sth like docker_flow.py to avoid confusing Python imports 😄
Yup, I saw that. That's when I went ahead just to see if I can get up and running with the Process block. Funny thing is - I don't I actually don't have a file named docker.py. Prefect has... https://github.com/PrefectHQ/prefect/blob/main/src/prefect/infrastructure/docker.py#L78
So when I found this "new" issue (different than Friday's), I decided to just pause and revisit when I have a POC up
Created an GH issue for the "new" issue https://github.com/PrefectHQ/prefect/issues/6906 Now I can check out the Docker stuff again.
😖 1
🙏 1
@Stéphan Taljaard have you tried running this without poetry? e.g. in a conda environment or even from CI/CD? my intuition is that poetry may confuse Prefect about the paths somehow
I'll try that out tomorrow. Though I don't think my workflow is way out of the ordinary?
I just reproduced, you are 100% right there is some bug here, it works in a usual setting on GCS but subpaths don't. Thanks for reporting this!