Ryan Sattler
08/04/2021, 6:11 AMprefect agent kubernetes start
. When I kick off the hello world flow, this indeed starts a job container on my desktop k8s which appears to start up fine and have the values that I configured (eg correct image). However it stays in a “running” state for several minutes until it errors out as follows: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5455f07710>: Failed to establish a new connection: [Errno -3] Tempor
Here’s the code for my flow:
import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun
@task
def hello_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello world!")
flow = Flow("hello-flow2", tasks=[hello_task])
# flow.run() We could run our flow locally using the flow's run method but we'll be running this from Cloud!
flow.run_config = KubernetesRun(
env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
image="prefecthq/prefect:latest",
labels=[]
)
flow.register(project_name="tester")
ModuleNotFoundError: No module named '/Users/ryan'
I don’t understand what’s happening there? (code is as above)No module named '/Users/ryan' │
│ Traceback (most recent call last): │
│ File "/usr/local/bin/prefect", line 8, in <module> │
│ sys.exit(cli()) │
│ File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__ │
│ return self.main(*args, **kwargs) │
│ File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main │
│ rv = self.invoke(ctx) │
│ File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke │
│ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke │
│ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke │
│ return ctx.invoke(self.callback, **ctx.params) │
│ File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke │
│ return callback(*args, **kwargs) │
│ File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 96, in flow_run │
│ raise exc │
│ File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 73, in flow_run │
│ flow = storage.get_flow(flow_data.name) │
│ File "/usr/local/lib/python3.7/site-packages/prefect/storage/local.py", line 103, in get_flow │
│ module_str=flow_location, flow_name=flow_name │
│ File "/usr/local/lib/python3.7/site-packages/prefect/utilities/storage.py", line 126, in extract_flow_from_module │
│ module = importlib.import_module(mod_name) │
│ File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module │
│ return _bootstrap._gcd_import(name[level:], package, level) │
│ File "<frozen importlib._bootstrap>", line 1006, in _gcd_import │
│ File "<frozen importlib._bootstrap>", line 983, in _find_and_load │
│ File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked │
│ File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed │
│ File "<frozen importlib._bootstrap>", line 1006, in _gcd_import │
│ File "<frozen importlib._bootstrap>", line 983, in _find_and_load │
│ File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked │
│ File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed │
│ File "<frozen importlib._bootstrap>", line 1006, in _gcd_import │
│ File "<frozen importlib._bootstrap>", line 983, in _find_and_load │
│ File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked │
│ ModuleNotFoundError: No module named '/Users/ryan'
Kevin Kho
08/04/2021, 2:45 PMLocalStorage
and the default behavior is that it will save it in the home directory. The issue here is that the KubernetesRun
created a new pod and then looks for it locally there, but doesn’t find the Flow.
A lot of users use DockerStorage
with KubernetesRun
where they package everything up in a container and then pass it to the KubernetesRun
so that the flows can be pulled from inside the container. You can also use something like S3
storage or Git
storage and Prefect will pull the flow down and run it on top of the image specified (prefecthq:prefect
) in your caseRyan Sattler
08/05/2021, 10:27 PM