Thread
#prefect-community
    Michael Hadorn

    Michael Hadorn

    1 year ago
    Hi there Sadly i didn't understand really the docker-run and gitlab-storage concept. 😞 I built an image on top of the prefect image (prefecthq/prefect:0.14.6-python3.8) and installed my python deps successfully. In my flow, I added to parts:
    with Flow(
        prefect.config.general.flow_name
        , executor=LocalDaskExecutor()
        , storage=GitLab(
            host="<https://gitlab.xxx.ch/>"
            , repo="cdwh/cdwh-prefect"
            , path="cdwh_flow.py"
            , ref="master"
            , secrets=["GITLAB_ACCESS_TOKEN"]
        )
        , run_config=DockerRun(
            image="mrjck/cdwh-flow:1.0"
        )
        ,...
    Then I successfully registered the flow to the backend. If I let them run, the prefect docker agent deploying the flow but crashed immediately with the following message:
    404 Client Error for <http+docker://localhost/v1.40/images/create?tag=1.0&fromImage=mrjck%2Fcdwh-flow>: Not Found ("manifest for mrjck/cdwh-flow:1.0 not found: manifest unknown: manifest unknown")
    If I run 'docker images' the used images is there with the correct tag. How does my Image know, how the storage is pulled? UPDATE: Was solved the next day. If not, looks like this: https://github.com/PrefectHQ/prefect/issues/2781 Try to start the agent with: --no-docker-interface
    I use the same flow to register and run the code (with some arguments). Is this the idea? So when is the image pulling the code? (are the any files from the repository included?)
    nicholas

    nicholas

    1 year ago
    Hi @Michael Hadorn - did the agent crash as it was deploying the flow or did it do so as you were starting it?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    @nicholas When I started what? 😅 I started it with the UI. Saw the agent deploying it, then the crash. The log message was already in the run log in the UI.
    nicholas

    nicholas

    1 year ago
    Gotcha, so the agent was already running as expected and was polling for work, right?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    yes. I think, the docker agent deployed the flow run, then the process tried to pull the image for this task and this crashes
    nicholas

    nicholas

    1 year ago
    Ok that narrows it down a bit, thank you.
    What happens when you run
    docker pull mrjck/cdwh-flow:1.0
    ?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    it will load. but anyway, I hope I could also use local images.
    poc on rnddkrpocwe01 in ~ via C base
    > docker pull mrjck/cdwh-flow:1.0
    1.0: Pulling from mrjck/cdwh-flow
    a076a628af6f: Already exists
    a36ca90be64c: Already exists
    44f7d13c37e7: Already exists
    b0d4acfb9127: Already exists
    1b9127658757: Pull complete
    299ecd49d09a: Pull complete
    d12193da5a01: Pull complete
    64f3d8908a2d: Pull complete
    f2d8b427df9a: Pull complete
    243d98899535: Pull complete
    Digest: sha256:6daedcbd339b76f07b627a0577678d38c1a5aa755b70e0ca000a85335608a329
    Status: Downloaded newer image for mrjck/cdwh-flow:1.0
    <http://docker.io/mrjck/cdwh-flow:1.0|docker.io/mrjck/cdwh-flow:1.0>
    nicholas

    nicholas

    1 year ago
    Hm ok, I'm a little stumped; that error is usually caused by the daemon being unable to access the image for some reason; assuming you're running these all on the same machine I'm not sure why the daemon would be able to pull the image in one scenario but not another.
    Michael Hadorn

    Michael Hadorn

    1 year ago
    Ok. But I really don't exactly how it's working anyway. In our case, we build a flow tasks based on external database records. This flow will be registered then. If there are changes in the database, I guess they will not be loaded, because this flow definition is static. I have to reregister the flow, right? For develop purpose, I would like to run the jobs for a subselection of these records, and wanna specify this as a param. But as I understand, I can not really do this in a existing flow. I think I have to register always a new one (with these tasks i need), right? So maybe best to achive this, is to have another new flow, who's registering basics flows. Then the developer can choose in the UI, for which objects the flow should be build.
    Michael Adkins

    Michael Adkins

    1 year ago
    Where you just showed
    docker pull ...
    logs -- is that on the same machine as the agent?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    BTW: can we rent some of you for a call / screen share support, if yes what would be the price?
    yes, i did the docker pull on the same machine
    nicholas

    nicholas

    1 year ago
    Yes to your first question @Michael Hadorn - if you want to DM me your email I can have someone reach out to you about that
    Michael Adkins

    Michael Adkins

    1 year ago
    Can you post the full log from
    Deploying flow run ...
    to the error from the agent?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    @nicholas Ok cool, have to ask my boss first. Will ask next week.
    @Michael Adkins Also this I will continue on monday. Best thanks for your awesome support!
    Ok. The first issue was solved tody. Read something about docker.hub problems. But now, i got a more common error:
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3298f358b0>: Failed to establish a new connection: [Errno 111] Connection refused'))>
    then i found out, about the
    --show-flow-logs
    Seems to have something todo with the network. My agent is started like this:
    prefect agent docker start -a <http://localhost:9001/graphql> -l windows --show-flow-logs --network prefect-server
    the --network attr does not help the ufw is disabled. any help?
    nicholas

    nicholas

    1 year ago
    Hm are you able to ping that endpoint to see if it's available? (Or open up a browser window)
    Michael Hadorn

    Michael Hadorn

    1 year ago
    It's while the run in the docker agent. after pulling the image. looks quite similar like this: https://prefect-community.slack.com/archives/CL09KU1K7/p1611719846242900?thread_ts=1611604594.167500&amp;cid=CL09KU1K7 but in my case, ufw is disabled.
    nicholas

    nicholas

    1 year ago
    Gotcha - what do you get back when you run
    curl <http://localhost:9001/graphql>
    ?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    on the host?
    GET query missing.
    Looks valid. Also the UI is working. if i change the -a attr, the docker agent will not start. The flow container will not been created.
    nicholas

    nicholas

    1 year ago
    And you're starting the agent on the same machine, right?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    Yes.
    In my case, i had to shift the ports above 9000. So my graphql is listening on 9001 the agent can start because i define the endpoint with the -a attr. but the error message, looks like wrong. HTTPConnectionPool(host='host.docker.internal', port=4200) i think there should be the correct port. in my case 9001
    but i'm not sure, where this port is set
    i can overwrite it with:
    export PREFECT__SERVER__PORT=9001
    then i got:
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', *port=9001*): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd5a16808b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    nicholas

    nicholas

    1 year ago
    Hm - can you provide some more details about the system, like OS/version?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    Ubuntu 18.04.5 LTS prefect==0.14.6
    I removed the --network attr (because it seems to go over host: host.docker.internal) Now i got bad request: GRAPHQL_VALIDATION_FAILED: Cannot query field "secret_value" on type "Query".
    nicholas

    nicholas

    1 year ago
    Perfect - it looks like at least the networking portion is correct. Can you try stopping the agent, running
    prefect backend server
    , and starting the agent again?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    also stopping the server for this backend change? before i had to create the tenant with:
    prefect server create-tenant --name default --slug default
    if only stop the agent, it does not work
    nicholas

    nicholas

    1 year ago
    Hm no you shouldn't need to stop the agent, that error tells us the agent is looking for secrets, which aren't available in the server graphql schema
    Sorry, when is that bad request coming up? On agent startup?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    no, still after the flow started:
    [2021-02-08 17:03:48,313] INFO - agent | Waiting for flow runs...
    [2021-02-08 17:04:04,053] INFO - agent | Found 1 flow run(s) to submit for execution.
    [2021-02-08 17:04:04,117] INFO - agent | Deploying flow run e77596a7-7c40-44dd-af20-7e1634a63f01
    [2021-02-08 17:04:04,119] INFO - agent | Pulling image mrjck/cdwh-flow:1.0...
    [2021-02-08 17:04:05,386] INFO - agent | Successfully pulled image mrjck/cdwh-flow:1.0...
    400 Client Error: Bad Request for url: <http://host.docker.internal:9001/graphql>
    
    The following error messages were provided by the GraphQL server:
    
        GRAPHQL_VALIDATION_FAILED: Cannot query field "secret_value" on type "Query".
    
    The GraphQL query was:
    
        query($name: String!) {
                    secret_value(name: $name)
        }
    
    The passed variables were:
    
        {"name": "GITLAB_ACCESS_TOKEN"}
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/client/secrets.py", line 137, in get
        value = secrets[self.name]
    KeyError: 'GITLAB_ACCESS_TOKEN'
    Hm. looks like the gitlab_access_token is missing. but is exported before
    export PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN='xxx'
    nicholas

    nicholas

    1 year ago
    Ok then yes these are separate issues - do you
    use_local_secrets
    set to
    True
    in your config?
    Michael Hadorn

    Michael Hadorn

    1 year ago
    where? in the configuration of my flow?
    nicholas

    nicholas

    1 year ago
    You should be able to set that in
    ~/prefect/config.toml
    on your host machine
    Michael Hadorn

    Michael Hadorn

    1 year ago
    sorry, i do not understand exactly. also first: no I didn't set this prop. but where do i have to set this? is this, right:
    [cloud]
       use_local_secrets = true
    or setting it via env:
    PREFECT__CLOUD__USE_LOCAL_SECRETS = true
    but i use the docker agent, so i'm sadly not sure to set this in the image itself, or before starting the docker agent (and in this context)
    (both is not working)
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! Looks like you're encountering https://github.com/PrefectHQ/prefect/issues/4051
    use_local_secrets
    only affects Prefect Cloud usage, with Prefect Server it is not relevant as local secrets are the only option for secrets.
    If I misunderstand and you're using Prefect Cloud, then you should note that local secrets are explicitly disabled from the DockerAgent because the secrets are not passed from your local config into the docker container.
    Michael Hadorn

    Michael Hadorn

    1 year ago
    Perfect. That's solve my issue! Thanks a lot.