Hi there Sadly i didn't understand really the docker-run and gitlab-storage concept. 😞 I built an image on top of the prefect image (prefecthq/prefect:0.14.6-python3.8) and installed my python deps successfully. In my flow, I added to parts:

with Flow(
    prefect.config.general.flow_name
    , executor=LocalDaskExecutor()
    , storage=GitLab(
        host="<https://gitlab.xxx.ch/>"
        , repo="cdwh/cdwh-prefect"
        , path="cdwh_flow.py"
        , ref="master"
        , secrets=["GITLAB_ACCESS_TOKEN"]
    )
    , run_config=DockerRun(
        image="mrjck/cdwh-flow:1.0"
    )
    ,...

Then I successfully registered the flow to the backend. If I let them run, the prefect docker agent deploying the flow but crashed immediately with the following message:

404 Client Error for <http+docker://localhost/v1.40/images/create?tag=1.0&fromImage=mrjck%2Fcdwh-flow>: Not Found ("manifest for mrjck/cdwh-flow:1.0 not found: manifest unknown: manifest unknown")

If I run 'docker images' the used images is there with the correct tag. How does my Image know, how the storage is pulled? UPDATE: Was solved the next day. If not, looks like this: https://github.com/PrefectHQ/prefect/issues/2781 Try to start the agent with: --no-docker-interface

Hi @Michael Hadorn - did the agent crash as it was deploying the flow or did it do so as you were starting it?

@nicholas When I started what? 😅 I started it with the UI. Saw the agent deploying it, then the crash. The log message was already in the run log in the UI.

Gotcha, so the agent was already running as expected and was polling for work, right?

yes. I think, the docker agent deployed the flow run, then the process tried to pull the image for this task and this crashes

Ok that narrows it down a bit, thank you.

it will load. but anyway, I hope I could also use local images.

poc on rnddkrpocwe01 in ~ via C base
> docker pull mrjck/cdwh-flow:1.0
1.0: Pulling from mrjck/cdwh-flow
a076a628af6f: Already exists
a36ca90be64c: Already exists
44f7d13c37e7: Already exists
b0d4acfb9127: Already exists
1b9127658757: Pull complete
299ecd49d09a: Pull complete
d12193da5a01: Pull complete
64f3d8908a2d: Pull complete
f2d8b427df9a: Pull complete
243d98899535: Pull complete
Digest: sha256:6daedcbd339b76f07b627a0577678d38c1a5aa755b70e0ca000a85335608a329
Status: Downloaded newer image for mrjck/cdwh-flow:1.0
<http://docker.io/mrjck/cdwh-flow:1.0|docker.io/mrjck/cdwh-flow:1.0>

Hm ok, I'm a little stumped; that error is usually caused by the daemon being unable to access the image for some reason; assuming you're running these all on the same machine I'm not sure why the daemon would be able to pull the image in one scenario but not another.

Ok. But I really don't exactly how it's working anyway. In our case, we build a flow tasks based on external database records. This flow will be registered then. If there are changes in the database, I guess they will not be loaded, because this flow definition is static. I have to reregister the flow, right? For develop purpose, I would like to run the jobs for a subselection of these records, and wanna specify this as a param. But as I understand, I can not really do this in a existing flow. I think I have to register always a new one (with these tasks i need), right? So maybe best to achive this, is to have another new flow, who's registering basics flows. Then the developer can choose in the UI, for which objects the flow should be build.

Where you just showed

docker pull ...

logs -- is that on the same machine as the agent?

BTW: can we rent some of you for a call / screen share support, if yes what would be the price?

Yes to your first question @Michael Hadorn - if you want to DM me your email I can have someone reach out to you about that

Can you post the full log from

Deploying flow run ...

to the error from the agent?

@nicholas Ok cool, have to ask my boss first. Will ask next week.

Hm are you able to ping that endpoint to see if it's available? (Or open up a browser window)

It's while the run in the docker agent. after pulling the image. looks quite similar like this: https://prefect-community.slack.com/archives/CL09KU1K7/p1611719846242900?thread_ts=1611604594.167500&cid=CL09KU1K7 but in my case, ufw is disabled.

Gotcha - what do you get back when you run

curl <http://localhost:9001/graphql>

?

on the host?

GET query missing.

Looks valid. Also the UI is working. if i change the -a attr, the docker agent will not start. The flow container will not been created.

And you're starting the agent on the same machine, right?

Yes.

Hm - can you provide some more details about the system, like OS/version?

Ubuntu 18.04.5 LTS prefect==0.14.6

Perfect - it looks like at least the networking portion is correct. Can you try stopping the agent, running

prefect backend server

, and starting the agent again?

also stopping the server for this backend change? before i had to create the tenant with:

prefect server create-tenant --name default --slug default

if only stop the agent, it does not work

Hm no you shouldn't need to stop the agent, that error tells us the agent is looking for secrets, which aren't available in the server graphql schema

no, still after the flow started:

[2021-02-08 17:03:48,313] INFO - agent | Waiting for flow runs...
[2021-02-08 17:04:04,053] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-02-08 17:04:04,117] INFO - agent | Deploying flow run e77596a7-7c40-44dd-af20-7e1634a63f01
[2021-02-08 17:04:04,119] INFO - agent | Pulling image mrjck/cdwh-flow:1.0...
[2021-02-08 17:04:05,386] INFO - agent | Successfully pulled image mrjck/cdwh-flow:1.0...
400 Client Error: Bad Request for url: <http://host.docker.internal:9001/graphql>

The following error messages were provided by the GraphQL server:

    GRAPHQL_VALIDATION_FAILED: Cannot query field "secret_value" on type "Query".

The GraphQL query was:

    query($name: String!) {
                secret_value(name: $name)
    }

The passed variables were:

    {"name": "GITLAB_ACCESS_TOKEN"}

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/secrets.py", line 137, in get
    value = secrets[self.name]
KeyError: 'GITLAB_ACCESS_TOKEN'

Hm. looks like the gitlab_access_token is missing. but is exported before

Ok then yes these are separate issues - do you

use_local_secrets

set to

True

in your config?

where? in the configuration of my flow?

You should be able to set that in

~/prefect/config.toml

on your host machine

sorry, i do not understand exactly. also first: no I didn't set this prop. but where do i have to set this? is this, right:

[cloud]
   use_local_secrets = true

or setting it via env:

PREFECT__CLOUD__USE_LOCAL_SECRETS = true

but i use the docker agent, so i'm sadly not sure to set this in the image itself, or before starting the docker agent (and in this context)

Hi! Looks like you're encountering https://github.com/PrefectHQ/prefect/issues/4051

Perfect. That's solve my issue! Thanks a lot.