Horatiu Bota

    Horatiu Bota

    4 months ago
    hi community! i'm trying to run a flow on AWS ECS using the ECS run config (https://docs.prefect.io/api/latest/run_configs.html#ecsrun) but my flow get stuck in a submitted state (already checked-out https://discourse.prefect.io/t/why-is-my-flow-stuck-in-a-submitted-state/201). I can see in the AWS console that the ECS task fails because the run command is
    Command ["/bin/sh","-c","prefect execute flow-run"]
    (however i need to run prefect in bash with a conda environment activated). Is there any way to configure the ECS task to remove the
    ["/bin/sh/", "-c",]
    before the prefect command? My container is configured with the appropriate entrypoint, runs bash and activates conda at startup, but
    /bin/sh -c
    undoes that setup
    tried setting something like this in the ECSRun setup:
    flow.run_config = ECSRun(
        task_definition_path="s3://<bucket>/task.yml",
        run_task_kwargs={
            "cluster": "test-prefect-cluster", 
            "overrides": {
            "containerOverrides": [{
                    "name": "flow",
                    "command": ["echo hello", "prefect execute flow-run"]
                }]
            }
        }
    )
    but no success
    Kevin Kho

    Kevin Kho

    4 months ago
    You shouldn’t need to set the entrypoint. The agent will do that for you. I think the issue here is more of activating conda is tricky. Do you need conda? The container should already provide isolation
    Horatiu Bota

    Horatiu Bota

    4 months ago
    conda is active in the container (i followed the article you linked above 😅), so running:
    docker run -it image-tag
    locally starts a shell with the conda env activated
    Kevin Kho

    Kevin Kho

    4 months ago
    Ah ok. There must be something wrong with Flow spinup. Check this thread and there is a code example you can use to add logs to the Flow to get better error messages
    Horatiu Bota

    Horatiu Bota

    4 months ago
    thanks! already read through that and have logs enabled - the only log message i'm getting is
    /bin/sh: /bin/sh: cannot execute binary file
    Kevin Kho

    Kevin Kho

    4 months ago
    Ok I re-read again. I know what you are saying now. I am actually confused where
    "/bin/sh"
    is added because I think we just add the
    prefect execute flow run
    here
    Oh my bad. It’s hardcoded here
    I dont think you can override this on the entrypoint though because this is a command added on top of that
    Horatiu Bota

    Horatiu Bota

    4 months ago
    hm, ok, thank you!