Serge Tarkovski

    Serge Tarkovski

    6 months ago
    Hi all! I followed the "Running flows in Docker" tutorial for 2.0 (
    prefect==2.0b2
    , Python 3.9, Linux): • created an S3 bucket and a storage (isn't it too much for a local run?) • started a local API server and configured PREFECT_API_URL • created a queue • started an agent for that queue • created a deployment with DockerFlowRunner and no parameters Then I ran the deployment and the agent output wasn't very informative, see below. In the UI the flow run "dag-sample/caped-antelope" is shown as failed without any logs available. Anyone can explain?
    $ prefect agent start 6aa0c2b2-e895-4b8c-aed3-8bef5b2c88ab
    Starting agent connected to <http://127.0.0.1:4200/api>...
    
      ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
     | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
     |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
     |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|
    
    
    Agent started! Looking for work from queue '6aa0c2b2-e895-4b8c-aed3-8bef5b2c88ab'...
    13:29:31.446 | INFO    | prefect.agent - Submitting flow run '8eb8aa0b-82da-4081-9a42-aae1f22b0525'
    /home/tarkovskyi/miniconda3/envs/prefect_exp39/lib/python3.9/site-packages/prefect/flow_runners.py:697: UserWarning: `host.docker.internal` could not be automatically resolved to your local ip address. This feature is not supported on Docker Engine v19.3.15, upgrade to v20.10.0+ if you encounter issues.
      warnings.warn(
    13:29:32.215 | INFO    | prefect.flow_runner.docker - Flow run 'caped-antelope' has container settings = {'image': 'prefecthq/prefect:2.0b2-python3.9', 'network': None, 'network_mode': 'host', 'command': ['python', '-m', 'prefect.engine', '8eb8aa0b-82da-4081-9a42-aae1f22b0525'], 'environment': {'PREFECT_API_URL': '<http://127.0.0.1:4200/api>'}, 'auto_remove': False, 'labels': {'io.prefect.flow-run-id': '8eb8aa0b-82da-4081-9a42-aae1f22b0525'}, 'extra_hosts': {}, 'name': 'caped-antelope', 'volumes': []}
    13:29:33.547 | INFO    | prefect.agent - Completed submission of flow run '8eb8aa0b-82da-4081-9a42-aae1f22b0525'
    13:29:33.584 | INFO    | prefect.flow_runner.docker - Flow run container 'caped-antelope' has status 'running'
    13:29:44.923 | INFO    | prefect.flow_runner.docker - Flow run container 'caped-antelope' has status 'exited'
    Anna Geller

    Anna Geller

    6 months ago
    It's likely that your flow run container exists due to missing AWS credentials within the container which are required to pull the flow from S3. Check this thread showing how you can mount credentials to your container on the DockerFlowRunner:https://prefect-community.slack.com/archives/CL09KU1K7/p1647435037255209?thread_ts=1647434164.837499&amp;cid=CL09KU1K7 And thanks for the feedback on storage - we are working on making the default storage more intuitive and easier to use.
    Serge Tarkovski

    Serge Tarkovski

    6 months ago
    actually, if there's another simpler storage type I can use, I'd be happy to
    I see there's a FileStorage (maybe I'm not 100% correct in the name) not listed in the docs, tried it as well but didn't succeed in mounting my local folder with the storage into the container
    Anna Geller

    Anna Geller

    6 months ago
    I think S3 is the easiest for now but we're working on it 🙂 you'll definitely see more updates on that in the next couple of weeks