Hello! I am trying to run a simple flow using a Do...
# prefect-community
p
Hello! I am trying to run a simple flow using a DockerContainer infrastructue and minio storage block. When the deployment is created, the flow is uploaded to minio but when I run the flow I get an error. Any ideas? Flow and error in thread
1
Copy code
from prefect import flow, get_run_logger
from prefect.deployments import Deployment
from prefect.filesystems import RemoteFileSystem
from prefect.infrastructure import DockerContainer


@flow
def my_docker_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello from Docker!")


minio_block = RemoteFileSystem(
    basepath="<s3://prefect-flows/flows>",
    key_type="hash",
    settings=dict(
        use_ssl = False,
        key = "blablabla",
        secret = "blablabla",
        client_kwargs = dict(endpoint_url = "<http://minio:9000>")
    ),
)
minio_block.save("minio", overwrite=True)


deployment = Deployment.build_from_flow(
    name="docker-example",
    flow=my_docker_flow,
    storage=RemoteFileSystem.load('minio'),
    infrastructure=DockerContainer(
        image = 'prefect-orion:2.1.1',
        image_pull_policy = 'IF_NOT_PRESENT',
        networks = ['prefect'],
        env = {
            "USE_SSL": False,
            "AWS_ACCESS_KEY_ID": "blablabla",
            "AWS_SECRET_ACCESS_KEY": "blablabla",
            "ENDPOINT_URL": '<http://minio:9000>',
        }
    ),
)
deployment.apply()
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 839, in exec_module
  File "<frozen importlib._bootstrap_external>", line 975, in get_code
  File "<frozen importlib._bootstrap_external>", line 1032, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/flows/test_flow.py'
in the docs I see references to
packager
but I have been told they are not supported in 2.0 anymore?
a
It looks like you are running minio itself in Docker and your flow run container can't access that I'd recommend switching to Process infrastructure block if you want to use minio, or switch to S3 storage block if you want to use DockerContainer block
p
it is strange bc this same setup used to work with beta8
this is the full traceback I get:
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 839, in exec_module
  File "<frozen importlib._bootstrap_external>", line 975, in get_code
  File "<frozen importlib._bootstrap_external>", line 1032, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/flows/test_flow.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 246, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 105, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 73, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(import_object, str(import_path))
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 193, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at '/flows/test_flow.py' encountered an exception
if I stop minio i get a completely different traceback:
Copy code
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "<http://minio:9000/prefect-flows?list-type=2&prefix=flows%2F&delimiter=&encoding-type=url>"
a
My recommendation is still unchanged :)
a
Glad the PR resolved the issue for you. what I was implying is that if your minio is running in a container, there might be some extra work needed to set up network etc to make that work with your flow run container