https://prefect.io logo
Title
c

Chris Goddard

08/30/2020, 3:45 PM
Hey folks - I'm having issues getting a docker agent to run flows locally. I'm using prefect cloud - I have confirmed that I can run a non-docker flow locally - and trigger it from the cloud UI (so
backend
is correctly set and the runner token is available as an environment variable). however, when I give the flow docker storage and spin up a docker agent, nothing happens when I try to trigger a flow from the ui - no errors, it's just like it's not receiving any instructions from prefect cloud. I am working in WSL2 (widows linux subsystem) - which creates all kinds of hellish networking issues (classic windows) - but I've confirmed that docker is working and I've run the docker image that was created for my flow and run the flow manually within the container by unpickling and running
flow.run
The on thing I thought it might be was failure to connect to the docker daemon (in case wsl ran it somewhere else) but I've confirmed that it's running at unix:///var/run/docker.sock (I think earlier versions of WSL had an issue but I don't think that's what's going on). What else could I try? any suggestions?
prefect diagnostics
output:
{
  "config_overrides": {
    "cloud": {
      "agent": {
        "auth_token": true
      }
    },
    "context": {
      "secrets": false
    }
  },
  "env_vars": [
    "PREFECT__CLOUD__AGENT__AUTH_TOKEN"
  ],
  "system_information": {
    "platform": "Linux-4.19.104-microsoft-standard-x86_64-with-glibc2.29",
    "prefect_version": "0.13.4",
    "python_version": "3.8.1"
  }
}
n

nicholas

08/30/2020, 3:54 PM
Hi @Chris Goddard - do your flows enter a
Scheduled
State in the UI but never leave it? Or do they get
Submitted
?
c

Chris Goddard

08/30/2020, 3:56 PM
yeah I think they just stay in
Scheduled
and then I get a message saying they're behind schedule - so they never run
n

nicholas

08/30/2020, 3:58 PM
Got it! The first thing to check is that the labels on your flow and agent are matching. All the labels your flow has need to exist on your agent as well.
c

Chris Goddard

08/30/2020, 4:00 PM
oh - I didn't know that was a consideration - let me try that
ok progress - I got an error message in the UI!
400 Client Error: Bad Request ("invalid IP address in add-host: """)
ERROR - agent | Error while deploying flow: APIError(HTTPError('400 Client Error: Bad Request for url: <http+docker://localhost/v1.40/containers/create'>))
n

nicholas

08/30/2020, 4:06 PM
Oh interesting, I’ve not seen that before. It looks like your agent isn’t able to communicate with the docker daemon
c

Chris Goddard

08/30/2020, 4:09 PM
hmm. it could well be wsl related. it has all kinds of trouble with localhost. because... microsoft.
will try a few things and let you know if I find anything else. thanks for your help!
n

nicholas

08/30/2020, 4:11 PM
Yeah that sounds viable. Glad to help!
c

Chris Goddard

08/30/2020, 4:37 PM
Sorry @nicholas - this is my own ignorance with docker but do you know how to find out where docker's base url is. docker is working - but wsl has some funky setup with docker and unfortunately all my searching only finds other people with similar problems but no resolution. I see reference to "host.docker.internal" - but I don't know how/where to find that
n

nicholas

08/30/2020, 4:51 PM
Hm not off the top of my head but I think there’s a configuration file somewhere that’ll have it. Sorry I’m not super familiar with the windows environment ☹️
c

Chris Goddard

08/30/2020, 4:51 PM
haha - neither. definitely not my choice :S
n

nicholas

08/30/2020, 4:53 PM
Oh! You have a docker desktop app, right?
c

Chris Goddard

08/30/2020, 4:53 PM
yeah
n

nicholas

08/30/2020, 4:54 PM
I think there’s a setting for wsl to expose the daemon at a local host location
c

Chris Goddard

08/30/2020, 4:55 PM
I think it's this
n

nicholas

08/30/2020, 4:55 PM
Oh yup, they’ve got that in the manual 😆
I’m assuming that’s not working?
c

Chris Goddard

08/30/2020, 4:56 PM
no - at least, so far. one thing with windows is you never know if it's actually not working or you're just supposed to restart something to get it to work (I hate windows so much)
would you know of any way to test the localhost daemon? like a curl request or something? I'm really new at docker so I don't really understand the networking elements
n

nicholas

08/30/2020, 4:58 PM
Unfortunately I’m not sure either /:
c

Chris Goddard

08/30/2020, 4:58 PM
that's ok
hmm - I realize it might be a different issue than I thought. 400 error means the rest is being processed but something else is wrong (maybe not wsl specific). I found this issue https://github.com/PrefectHQ/prefect/issues/2361 that mentions the error
invalid IP address in add-host: """
I'm just using a simple docker storage object - with a base dockerfile and local_image=True. I'm not using a registry url. add-host is clearly part of how prefect is creating the container - is there some other docker networking configuration I might need?
get_docker_ip()
is returning null
SUCCESS! needed the
--no-docker-interface
flag on the prefect agent start command.
:upvote: 1
thanks for helping me work through it.
n

nicholas

08/30/2020, 6:12 PM
Oh that’s great! I’m glad it turned out to be something simple 😃