https://prefect.io logo
Title
r

Ryan Sattler

08/04/2021, 6:11 AM
Hi - I’ve been going through the Prefect tutorials using a free account. I’ve gotten the basic Hello World flow to run using the local agent, but now I’m trying to get it to run via the kubernetes agent (using MacOS desktop docker). I’ve run the command-line k8s agent as follows:
prefect agent kubernetes start
. When I kick off the hello world flow, this indeed starts a job container on my desktop k8s which appears to start up fine and have the values that I configured (eg correct image). However it stays in a “running” state for several minutes until it errors out as follows:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5455f07710>: Failed to establish a new connection: [Errno -3] Tempor
Here’s the code for my flow:
import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello world!")

flow = Flow("hello-flow2", tasks=[hello_task])

# flow.run() We could run our flow locally using the flow's run method but we'll be running this from Cloud!

flow.run_config = KubernetesRun(
    env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
    image="prefecthq/prefect:latest",
    labels=[]
)
flow.register(project_name="tester")
Hmm might be related to this issue: https://github.com/PrefectHQ/server/issues/25 though that was for Linux not MacOS 🤔
I am using version 3.3.1 of MacOS Docker for Desktop
Still having the same problem after upgrading to Docker Desktop 3.5.2
After resetting docker desktop to factory settings, I am now getting a different error:
ModuleNotFoundError: No module named '/Users/ryan'
I don’t understand what’s happening there? (code is as above)
Full traceback is:
No module named '/Users/ryan'                                                                                                                                                                                                                                                         │
│ Traceback (most recent call last):                                                                                                                                                                                                                                                    │
│   File "/usr/local/bin/prefect", line 8, in <module>                                                                                                                                                                                                                                  │
│     sys.exit(cli())                                                                                                                                                                                                                                                                   │
│   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__                                                                                                                                                                                                  │
│     return self.main(*args, **kwargs)                                                                                                                                                                                                                                                 │
│   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main                                                                                                                                                                                                      │
│     rv = self.invoke(ctx)                                                                                                                                                                                                                                                             │
│   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke                                                                                                                                                                                                   │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                           │
│   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke                                                                                                                                                                                                   │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                           │
│   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke                                                                                                                                                                                                   │
│     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                                                    │
│   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke                                                                                                                                                                                                    │
│     return callback(*args, **kwargs)                                                                                                                                                                                                                                                  │
│   File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 96, in flow_run                                                                                                                                                                                          │
│     raise exc                                                                                                                                                                                                                                                                         │
│   File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 73, in flow_run                                                                                                                                                                                          │
│     flow = storage.get_flow(flow_data.name)                                                                                                                                                                                                                                           │
│   File "/usr/local/lib/python3.7/site-packages/prefect/storage/local.py", line 103, in get_flow                                                                                                                                                                                       │
│     module_str=flow_location, flow_name=flow_name                                                                                                                                                                                                                                     │
│   File "/usr/local/lib/python3.7/site-packages/prefect/utilities/storage.py", line 126, in extract_flow_from_module                                                                                                                                                                   │
│     module = importlib.import_module(mod_name)                                                                                                                                                                                                                                        │
│   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module                                                                                                                                                                                                   │
│     return _bootstrap._gcd_import(name[level:], package, level)                                                                                                                                                                                                                       │
│   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import                                                                                                                                                                                                                     │
│   File "<frozen importlib._bootstrap>", line 983, in _find_and_load                                                                                                                                                                                                                   │
│   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked                                                                                                                                                                                                          │
│   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed                                                                                                                                                                                                        │
│   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import                                                                                                                                                                                                                     │
│   File "<frozen importlib._bootstrap>", line 983, in _find_and_load                                                                                                                                                                                                                   │
│   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked                                                                                                                                                                                                          │
│   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed                                                                                                                                                                                                        │
│   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import                                                                                                                                                                                                                     │
│   File "<frozen importlib._bootstrap>", line 983, in _find_and_load                                                                                                                                                                                                                   │
│   File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked                                                                                                                                                                                                          │
│ ModuleNotFoundError: No module named '/Users/ryan'
my impression is that it’s trying to store something in my home directory, but has somehow had that configured wrongly
k

Kevin Kho

08/04/2021, 2:45 PM
Hey @Ryan Sattler it looks like you don’t have Storage defined, which uses
LocalStorage
and the default behavior is that it will save it in the home directory. The issue here is that the
KubernetesRun
created a new pod and then looks for it locally there, but doesn’t find the Flow. A lot of users use
DockerStorage
with
KubernetesRun
where they package everything up in a container and then pass it to the
KubernetesRun
so that the flows can be pulled from inside the container. You can also use something like
S3
storage or
Git
storage and Prefect will pull the flow down and run it on top of the image specified (
prefecthq:prefect
) in your case
r

Ryan Sattler

08/05/2021, 10:27 PM
hmm thanks Kevin I’ll have a look into that