https://prefect.io logo
Channels
Powered by
m

Michael Hadorn

02/05/2021, 1:32 PM
Hi there Sadly i didn't understand really the docker-run and gitlab-storage concept. 😞 I built an image on top of the prefect image (prefecthq/prefect:0.14.6-python3.8) and installed my python deps successfully. In my flow, I added to parts:
Copy code
with Flow(
    prefect.config.general.flow_name
    , executor=LocalDaskExecutor()
    , storage=GitLab(
        host="<https://gitlab.xxx.ch/>"
        , repo="cdwh/cdwh-prefect"
        , path="cdwh_flow.py"
        , ref="master"
        , secrets=["GITLAB_ACCESS_TOKEN"]
    )
    , run_config=DockerRun(
        image="mrjck/cdwh-flow:1.0"
    )
    ,...
Then I successfully registered the flow to the backend. If I let them run, the prefect docker agent deploying the flow but crashed immediately with the following message: 
404 Client Error for <http+docker://localhost/v1.40/images/create?tag=1.0&fromImage=mrjck%2Fcdwh-flow>: Not Found ("manifest for mrjck/cdwh-flow:1.0 not found: manifest unknown: manifest unknown")
If I run 'docker images' the used images is there with the correct tag. How does my Image know, how the storage is pulled? UPDATE: Was solved the next day. If not, looks like this: https://github.com/PrefectHQ/prefect/issues/2781 Try to start the agent with: --no-docker-interface
I use the same flow to register and run the code (with some arguments). Is this the idea? So when is the image pulling the code? (are the any files from the repository included?)
n

nicholas

02/05/2021, 4:04 PM
Hi @Michael Hadorn - did the agent crash as it was deploying the flow or did it do so as you were starting it?
m

Michael Hadorn

02/05/2021, 4:06 PM
@nicholas When I started what? 😅 I started it with the UI. Saw the agent deploying it, then the crash. The log message was already in the run log in the UI.
n

nicholas

02/05/2021, 4:07 PM
Gotcha, so the agent was already running as expected and was polling for work, right?
m

Michael Hadorn

02/05/2021, 4:08 PM
yes. I think, the docker agent deployed the flow run, then the process tried to pull the image for this task and this crashes
n

nicholas

02/05/2021, 4:13 PM
Ok that narrows it down a bit, thank you.
What happens when you run 
docker pull mrjck/cdwh-flow:1.0
?
m

Michael Hadorn

02/05/2021, 4:21 PM
it will load. but anyway, I hope I could also use local images.
Copy code
poc on rnddkrpocwe01 in ~ via C base
> docker pull mrjck/cdwh-flow:1.0
1.0: Pulling from mrjck/cdwh-flow
a076a628af6f: Already exists
a36ca90be64c: Already exists
44f7d13c37e7: Already exists
b0d4acfb9127: Already exists
1b9127658757: Pull complete
299ecd49d09a: Pull complete
d12193da5a01: Pull complete
64f3d8908a2d: Pull complete
f2d8b427df9a: Pull complete
243d98899535: Pull complete
Digest: sha256:6daedcbd339b76f07b627a0577678d38c1a5aa755b70e0ca000a85335608a329
Status: Downloaded newer image for mrjck/cdwh-flow:1.0
<http://docker.io/mrjck/cdwh-flow:1.0|docker.io/mrjck/cdwh-flow:1.0>
n

nicholas

02/05/2021, 4:28 PM
Hm ok, I'm a little stumped; that error is usually caused by the daemon being unable to access the image for some reason; assuming you're running these all on the same machine I'm not sure why the daemon would be able to pull the image in one scenario but not another.
m

Michael Hadorn

02/05/2021, 4:41 PM
Ok. But I really don't exactly how it's working anyway. In our case, we build a flow tasks based on external database records. This flow will be registered then. If there are changes in the database, I guess they will not be loaded, because this flow definition is static. I have to reregister the flow, right? For develop purpose, I would like to run the jobs for a subselection of these records, and wanna specify this as a param. But as I understand, I can not really do this in a existing flow. I think I have to register always a new one (with these tasks i need), right? So maybe best to achive this, is to have another new flow, who's registering basics flows. Then the developer can choose in the UI, for which objects the flow should be build.
z

Zanie

02/05/2021, 4:42 PM
Where you just showed 
docker pull ...
logs -- is that on the same machine as the agent?
m

Michael Hadorn

02/05/2021, 4:42 PM
BTW: can we rent some of you for a call / screen share support, if yes what would be the price?
👍 1
yes, i did the docker pull on the same machine
n

nicholas

02/05/2021, 4:45 PM
Yes to your first question @Michael Hadorn - if you want to DM me your email I can have someone reach out to you about that
z

Zanie

02/05/2021, 4:45 PM
Can you post the full log from 
Deploying flow run ...
to the error from the agent?
m

Michael Hadorn

02/05/2021, 6:25 PM
@nicholas Ok cool, have to ask my boss first. Will ask next week.
🙏 1
@Zanie Also this I will continue on monday. Best thanks for your awesome support!
Ok. The first issue was solved tody. Read something about docker.hub problems. But now, i got a more common error: 
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3298f358b0>: Failed to establish a new connection: [Errno 111] Connection refused'))>
then i found out, about the 
--show-flow-logs
Seems to have something todo with the network. My agent is started like this: 
prefect agent docker start -a <http://localhost:9001/graphql> -l windows --show-flow-logs --network prefect-server
the --network attr does not help the ufw is disabled. any help?
n

nicholas

02/08/2021, 4:21 PM
Hm are you able to ping that endpoint to see if it's available? (Or open up a browser window)
m

Michael Hadorn

02/08/2021, 4:23 PM
It's while the run in the docker agent. after pulling the image. looks quite similar like this: https://prefect-community.slack.com/archives/CL09KU1K7/p1611719846242900?thread_ts=1611604594.167500&amp;cid=CL09KU1K7 but in my case, ufw is disabled.
n

nicholas

02/08/2021, 4:27 PM
Gotcha - what do you get back when you run 
curl <http://localhost:9001/graphql>
?
m

Michael Hadorn

02/08/2021, 4:32 PM
on the host? 
GET query missing.
Looks valid. Also the UI is working. if i change the -a attr, the docker agent will not start. The flow container will not been created.
n

nicholas

02/08/2021, 4:39 PM
And you're starting the agent on the same machine, right?
m

Michael Hadorn

02/08/2021, 4:40 PM
Yes.
In my case, i had to shift the ports above 9000. So my graphql is listening on 9001 the agent can start because i define the endpoint with the -a attr. but the error message, looks like wrong. HTTPConnectionPool(host='host.docker.internal', port=4200) i think there should be the correct port. in my case 9001
but i'm not sure, where this port is set
i can overwrite it with: 
export PREFECT__SERVER__PORT=9001
then i got: 
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', *port=9001*): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd5a16808b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
n

nicholas

02/08/2021, 4:55 PM
Hm - can you provide some more details about the system, like OS/version?
m

Michael Hadorn

02/08/2021, 4:56 PM
Ubuntu 18.04.5 LTS prefect==0.14.6
I removed the --network attr (because it seems to go over host: host.docker.internal) Now i got bad request: GRAPHQL_VALIDATION_FAILED: Cannot query field "secret_value" on type "Query".
n

nicholas

02/08/2021, 5:02 PM
Perfect - it looks like at least the networking portion is correct. Can you try stopping the agent, running 
prefect backend server
, and starting the agent again?
m

Michael Hadorn

02/08/2021, 5:05 PM
also stopping the server for this backend change? before i had to create the tenant with:
Copy code
prefect server create-tenant --name default --slug default
if only stop the agent, it does not work
n

nicholas

02/08/2021, 5:06 PM
Hm no you shouldn't need to stop the agent, that error tells us the agent is looking for secrets, which aren't available in the server graphql schema
Sorry, when is that bad request coming up? On agent startup?
m

Michael Hadorn

02/08/2021, 5:12 PM
no, still after the flow started:
Copy code
[2021-02-08 17:03:48,313] INFO - agent | Waiting for flow runs...
[2021-02-08 17:04:04,053] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-02-08 17:04:04,117] INFO - agent | Deploying flow run e77596a7-7c40-44dd-af20-7e1634a63f01
[2021-02-08 17:04:04,119] INFO - agent | Pulling image mrjck/cdwh-flow:1.0...
[2021-02-08 17:04:05,386] INFO - agent | Successfully pulled image mrjck/cdwh-flow:1.0...
400 Client Error: Bad Request for url: <http://host.docker.internal:9001/graphql>

The following error messages were provided by the GraphQL server:

    GRAPHQL_VALIDATION_FAILED: Cannot query field "secret_value" on type "Query".

The GraphQL query was:

    query($name: String!) {
                secret_value(name: $name)
    }

The passed variables were:

    {"name": "GITLAB_ACCESS_TOKEN"}

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/secrets.py", line 137, in get
    value = secrets[self.name]
KeyError: 'GITLAB_ACCESS_TOKEN'
Hm. looks like the gitlab_access_token is missing. but is exported before
export PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN='xxx'
n

nicholas

02/08/2021, 5:23 PM
Ok then yes these are separate issues - do you 
use_local_secrets
set to 
True
in your config?
m

Michael Hadorn

02/08/2021, 5:24 PM
where? in the configuration of my flow?
n

nicholas

02/08/2021, 5:27 PM
You should be able to set that in 
~/prefect/config.toml
on your host machine
m

Michael Hadorn

02/08/2021, 5:33 PM
sorry, i do not understand exactly. also first: no I didn't set this prop. but where do i have to set this? is this, right:
Copy code
[cloud]
   use_local_secrets = true
or setting it via env: 
PREFECT__CLOUD__USE_LOCAL_SECRETS = true
but i use the docker agent, so i'm sadly not sure to set this in the image itself, or before starting the docker agent (and in this context)
(both is not working)
z

Zanie

02/08/2021, 6:09 PM
Hi! Looks like you're encountering https://github.com/PrefectHQ/prefect/issues/4051
use_local_secrets
only affects Prefect Cloud usage, with Prefect Server it is not relevant as local secrets are the only option for secrets.
If I misunderstand and you're using Prefect Cloud, then you should note that local secrets are explicitly disabled from the DockerAgent because the secrets are not passed from your local config into the docker container.
m

Michael Hadorn

02/08/2021, 6:20 PM
Perfect. That's solve my issue! Thanks a lot.
2 Views
Powered by