Hi. At last I'm properly getting to 2.0! :success-kid: Seems I'm running into the same issue as Keit...

Stéphan Taljaard

09/16/2022, 1:44 PM

Hi. At last I'm properly getting to 2.0! success kid Seems I'm running into the same issue as Keith here Getting

ModuleNotFoundError: No module named 'gcsfs'

It's strange because the environment that my agent is running in has

gsfs

installed and I have

"EXTRA_PIP_PACKAGES": "gcsfs"

in my Docker Block... Any tips?

✅ 1

Stéphan Taljaard

09/16/2022, 1:45 PM

I've created deployments from a Python objects. I'm using a GCS Storage Block and Docker container infra block. I created the Docker Container block via the Cloud UI and used only defaults (i.e. left fields empty), except for Env=

Copy code

{
  "EXTRA_PIP_PACKAGES": "gcsfs",
  "GOOGLE_APPLICATION_CREDENTIALS": "/opt/prefect/dev-user-staljaard.json"
}

Volumes =

[ "/home/stephan/di/dev-user-staljaard.json:/opt/prefect/dev-user-staljaard.json" ]

Auto Remove = true

Anna Geller

09/16/2022, 2:00 PM

As an alternative, you can build a custom image, this might help -- I believe this could be an issue with the volume attachment. You can create a GitHub issue as well if this doesn't work (because it should) and you can tag me there, I'd be happy to reproduce both options (volumes and custom image)

Stéphan Taljaard

09/16/2022, 2:06 PM

Eventually I'm going to use a custom image anyway, but for testing things out I wanted to keep it simple 😅 👍 I'll try to get to it tonight

👍 2

blob attention gif 1

🙏 2

Stéphan Taljaard

09/19/2022, 5:22 AM

I couldn't get to it on Friday, but am busy drafting the issue. While doing so, testing again, I now get a lot of

Copy code

prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/xxx/workspaces/xxx/deployments/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>

Any issues currently with the API for which I should wait before testing again and completing the ticket? https://prefect.status.io/ seems clear.

✅ 1

Stéphan Taljaard

09/19/2022, 9:59 AM

I managed to get around this by manually deleting all flows & deployments Even in the Cloud GUI I got errors preventing deletion. I had to Ctrl + F5, after which I was able to delete all. I'll get back to the Docker problems.

Anna Geller

09/19/2022, 10:48 AM

Hi Stephan! How are things going on this? I can try to reproduce now - did you open an issue?

Anna Geller

09/19/2022, 10:50 AM

Prefect API is fine, just ran some flows - all good on that end

👍 1

Stéphan Taljaard

09/19/2022, 11:03 AM

I ran into other issues as well (flow runs failing when running using a Docker Container block - complaining about

AttributeError: module 'docker' has no attribute 'from_env'

I'm trying out a Subprocess block for now; need to make progress. I'll revisit gain and create a detailed issue.

👍 1

Anna Geller

09/19/2022, 11:08 AM

so I just did the following and it worked:

Copy code

import json
from prefect.infrastructure import DockerContainer
from prefect.filesystems import GCS

service_account_info = json.load(open("/Users/anna/sa.json"))
gcs = GCS(
    bucket_path="prefect-orion/flows",
    service_account_info=str(service_account_info),
)
gcs.save("dev", overwrite=True)

docker_block = DockerContainer(
    image="prefecthq/prefect:2-python3.10", env={"EXTRA_PIP_PACKAGES": "gcsfs"}
)
docker_block.save("dev", overwrite=True)

# CLI
pip install gcsfs
prefect deployment build -n dev -q dev -sb gcs/dev -ib docker-container/dev -a flows/healthcheck.py:healthcheck

prefect deployment run healthcheck/dev
prefect agent start -q dev

Anna Geller

09/19/2022, 11:09 AM

will now change to 2.4.0 image

Anna Geller

09/19/2022, 11:11 AM

googling your error, SO says: "I had pip installed docker-py, but the current version of the package is simply called "docker" on pypi."

Anna Geller

09/19/2022, 11:12 AM

could be package version mismatch - I'd recommend creating a new conda environment and installing latest Prefect version there, then starting the agent -- It should work then, but if it doesn't please create an issue and link here. Thanks Stephan!

Stéphan Taljaard

09/19/2022, 11:12 AM

I'll investigate again and create an issue, and give feedback 👍

Anna Geller

09/19/2022, 11:12 AM

wait, can it be that your flow file is called docker.py?

🙅 1

Anna Geller

09/19/2022, 11:13 AM

if so, please change it to sth like docker_flow.py to avoid confusing Python imports 😄

Anna Geller

09/19/2022, 11:13 AM

https://github.com/docker/docker-py/issues/1370#issuecomment-702372966

Stéphan Taljaard

09/19/2022, 11:14 AM

Yup, I saw that. That's when I went ahead just to see if I can get up and running with the Process block. Funny thing is - I don't I actually don't have a file named docker.py. Prefect has... https://github.com/PrefectHQ/prefect/blob/main/src/prefect/infrastructure/docker.py#L78

Stéphan Taljaard

09/19/2022, 11:17 AM

So when I found this "new" issue (different than Friday's), I decided to just pause and revisit when I have a POC up

Stéphan Taljaard

09/21/2022, 7:58 AM

Created an GH issue for the "new" issue https://github.com/PrefectHQ/prefect/issues/6906 Now I can check out the Docker stuff again.

😖 1

🙏 1

Anna Geller

09/21/2022, 10:55 AM

@Stéphan Taljaard have you tried running this without poetry? e.g. in a conda environment or even from CI/CD? my intuition is that poetry may confuse Prefect about the paths somehow

Stéphan Taljaard

09/21/2022, 11:06 AM

I'll try that out tomorrow. Though I don't think my workflow is way out of the ordinary?

Anna Geller

09/21/2022, 11:33 AM

I just reproduced, you are 100% right there is some bug here, it works in a usual setting on GCS but subpaths don't. Thanks for reporting this!

Stéphan Taljaard

09/21/2022, 11:39 AM

😊😊

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.