https://prefect.io logo
Powered by
Title
s

Stéphan Taljaard

09/16/2022, 1:44 PM
Hi. At last I'm properly getting to 2.0! 😒uccess-kid: Seems I'm running into the same issue as Keith here Getting 
ModuleNotFoundError: No module named 'gcsfs'
It's strange because the environment that my agent is running in has 
gsfs
installed and I have 
"EXTRA_PIP_PACKAGES": "gcsfs"
in my Docker Block... Any tips?
✅ 1
I've created deployments from a Python objects. I'm using a GCS Storage Block and Docker container infra block. I created the Docker Container block via the Cloud UI and used only defaults (i.e. left fields empty), except for Env= 
{
  "EXTRA_PIP_PACKAGES": "gcsfs",
  "GOOGLE_APPLICATION_CREDENTIALS": "/opt/prefect/dev-user-staljaard.json"
}
Volumes = 
[ "/home/stephan/di/dev-user-staljaard.json:/opt/prefect/dev-user-staljaard.json" ]
Auto Remove = true
a

Anna Geller

09/16/2022, 2:00 PM
As an alternative, you can build a custom image, this might help -- I believe this could be an issue with the volume attachment. You can create a GitHub issue as well if this doesn't work (because it should) and you can tag me there, I'd be happy to reproduce both options (volumes and custom image)
s

Stéphan Taljaard

09/16/2022, 2:06 PM
Eventually I'm going to use a custom image anyway, but for testing things out I wanted to keep it simple 😅 👍 I'll try to get to it tonight
👍 2
:blob-attention-gif: 1
:thank-you: 2
I couldn't get to it on Friday, but am busy drafting the issue. While doing so, testing again, I now get a lot of 
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/xxx/workspaces/xxx/deployments/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
Any issues currently with the API for which I should wait before testing again and completing the ticket? https://prefect.status.io/ seems clear.
✅ 1
I managed to get around this by manually deleting all flows & deployments Even in the Cloud GUI I got errors preventing deletion. I had to Ctrl + F5, after which I was able to delete all. I'll get back to the Docker problems.
a

Anna Geller

09/19/2022, 10:48 AM
Hi Stephan! How are things going on this? I can try to reproduce now - did you open an issue?
Prefect API is fine, just ran some flows - all good on that end
👍 1
s

Stéphan Taljaard

09/19/2022, 11:03 AM
I ran into other issues as well (flow runs failing when running using a Docker Container block - complaining about 
AttributeError: module 'docker' has no attribute 'from_env'
I'm trying out a Subprocess block for now; need to make progress. I'll revisit gain and create a detailed issue.
👍 1
a

Anna Geller

09/19/2022, 11:08 AM
so I just did the following and it worked: 
import json
from prefect.infrastructure import DockerContainer
from prefect.filesystems import GCS

service_account_info = json.load(open("/Users/anna/sa.json"))
gcs = GCS(
    bucket_path="prefect-orion/flows",
    service_account_info=str(service_account_info),
)
gcs.save("dev", overwrite=True)

docker_block = DockerContainer(
    image="prefecthq/prefect:2-python3.10", env={"EXTRA_PIP_PACKAGES": "gcsfs"}
)
docker_block.save("dev", overwrite=True)

# CLI
pip install gcsfs
prefect deployment build -n dev -q dev -sb gcs/dev -ib docker-container/dev -a flows/healthcheck.py:healthcheck

prefect deployment run healthcheck/dev
prefect agent start -q dev
will now change to 2.4.0 image
googling your error, SO says: "I had pip installed docker-py, but the current version of the package is simply called "docker" on pypi."
could be package version mismatch - I'd recommend creating a new conda environment and installing latest Prefect version there, then starting the agent -- It should work then, but if it doesn't please create an issue and link here. Thanks Stephan!
s

Stéphan Taljaard

09/19/2022, 11:12 AM
I'll investigate again and create an issue, and give feedback 👍
a

Anna Geller

09/19/2022, 11:12 AM
wait, can it be that your flow file is called docker.py?
🙅 1
if so, please change it to sth like docker_flow.py to avoid confusing Python imports 😄
https://github.com/docker/docker-py/issues/1370#issuecomment-702372966
s

Stéphan Taljaard

09/19/2022, 11:14 AM
Yup, I saw that. That's when I went ahead just to see if I can get up and running with the Process block. Funny thing is - I don't I actually don't have a file named docker.py. Prefect has... https://github.com/PrefectHQ/prefect/blob/main/src/prefect/infrastructure/docker.py#L78
So when I found this "new" issue (different than Friday's), I decided to just pause and revisit when I have a POC up
Created an GH issue for the "new" issue https://github.com/PrefectHQ/prefect/issues/6906 Now I can check out the Docker stuff again.
😖 1
:thank-you: 1
a

Anna Geller

09/21/2022, 10:55 AM
@Stéphan Taljaard have you tried running this without poetry? e.g. in a conda environment or even from CI/CD? my intuition is that poetry may confuse Prefect about the paths somehow
s

Stéphan Taljaard

09/21/2022, 11:06 AM
I'll try that out tomorrow. Though I don't think my workflow is way out of the ordinary?
a

Anna Geller

09/21/2022, 11:33 AM
I just reproduced, you are 100% right there is some bug here, it works in a usual setting on GCS but subpaths don't. Thanks for reporting this!
s

Stéphan Taljaard

09/21/2022, 11:39 AM
😊😊
#prefect-communityJoin Slack
Powered by