https://prefect.io logo
Powered by
Title
m

Michael C. Grant

07/22/2020, 3:49 PM
Hey folks, I'm experimenting with using a Dask Gateway cluster to handle workloads. Currently we're using a local Dask execution environment with success so we're good there. Does anyone happen to have a custom Dask Gateway worker with prefect preinstalled they'd be willing to share?
👋 2
👏 1
💯 3
j

Jim Crist-Harif

07/22/2020, 3:50 PM
Are you asking for a docker image with prefect and dask-gateway installed? What issues are you running into?
m

Michael C. Grant

07/22/2020, 4:03 PM
Well, I don't want to turn this into a dask-gateway support thread unless you really want to, but I've hit a couple of issues. • I tried to build a custom docker image from docker-gateway-base that includes dask-gateway, prefect, and some other packages of interest. It seems to work fine as a server but not as a worker—the pods keep terminating and restarting—and I'm struggling to properly diagnose that. • I tried to implement the 
extraConfig
approach in the dask-gateway docs to provide an option handler, but it looks like 
extraConfig
is by default a dict, so the example in the docs needs some modification (will file an issue). But even when I fix that, even the stock dask-gateway container chokes on it. Again, I know that's not enough context to really diagnose anything, and will follow up on more appropriate channels, but I figured that if someone already had a dask gateway worker with prefect installed it would be a great way to get started.
So the shorter answer is "yes I'm looking for a docker image with prefect and dask-gateway installed" 🙂
j

Jim Crist-Harif

07/22/2020, 4:05 PM
My guess is that's a versioning thing - distributed broke some stuff in new versions. If you take the stock dask-gateway images provided in the last release and install prefect without allowing conda to update dask/distributed I'd expect things to work. I'm about to push a new release of dask-gateway that fixes the compatibility issues, but that's not out yet.
I tried to implement the 
extraConfig
approach in the dask-gateway docs to provide an option handler, but it looks like 
extraConfig
is by default a dict, so the example in the docs needs some modification (will file an issue).
It can be either a dict or a str, both should work fine. If not, there's a bug in the helm chart.
m

Michael C. Grant

07/22/2020, 4:07 PM
Ah, that's good advice. I tried to align versions on some things (e.g., cloudpickle) but that's a better approach
It can be either a dict or a str, both should work fine. If not, there's a bug in the helm chart.
Helm seems to disagree, actually. At least later versions. It doesn't allow overriding a dict with a str
So I converted my string to a single-key dict and Helm was happy, but gateway is not.
j

Jim Crist-Harif

07/22/2020, 4:09 PM
pangeo is using the dict version just fine, perhaps a formatting error in your yaml/code? https://github.com/pangeo-data/helm-chart/blob/master/pangeo/values.yaml#L89-L107
m

Michael C. Grant

07/22/2020, 4:09 PM
extraConfig:
      config: |
        from dask_gateway_server.options import Options, Integer, Float, String
        def option_handler(options):
            return {
                "worker_cores": options.worker_cores,
                "worker_memory": "%fG" % options.worker_memory,
                "image": options.image,
            }
        c.Backend.cluster_options = Options(
            Integer("worker_cores", 2, min=1, max=4, label="Worker Cores"),
            Float("worker_memory", 4, min=1, max=8, label="Worker Memory (GiB)"),
            String("image", default="daskgateway/dask-gateway:latest", label="Image"),
            handler=option_handler,
        )
Thanks I'll take a look at that
j

Jim Crist-Harif

07/22/2020, 4:09 PM
both should work though (and used to). I'll try to find some time to take a look later.
m

Michael C. Grant

07/22/2020, 4:11 PM
Here's what Helm is saying, BTW: 
coalesce.go:196: warning: cannot overwrite table with non table for extraConfig (map[])
This might be Helm 3, but you're going to want to support Helm 3, it's very nice not to have to worry about tiller
j

Jim Crist-Harif

07/22/2020, 4:12 PM
That used to be fine in helm 3, will debug later.
m

Michael C. Grant

07/22/2020, 4:13 PM
https://github.com/pangeo-data/helm-chart/issues/21
@Jim Crist-Harif the options issue is a versioning problem: 
daskgateway/dask-gateway:latest
is not, in fact, the latest; it's 7 months old vs. 
0.7.1
, which is 3 months old
Once I used an explicit version tag all was well with the options.
j

Jim Crist-Harif

07/22/2020, 5:51 PM
Ah! sorry about that, we clearly need to automate our release process, I am fallible.
Glad you figured that out.
m

Michael C. Grant

07/22/2020, 5:52 PM
Indeed! The options functionality is great, so I'm glad it was such an easy fix. I'm not seeing a way around the Helm chart issue (i.e., needing to use a dict) but really that 's a doc fix as far as I'm concerned
👍 1
And I'm up and running with a customized container, too. I just installed a few extra things inside of the existing dask-gateway:0.7.1 image
j

Jim Crist-Harif

07/24/2020, 2:06 PM
I'm unable to reproduce, things work fine as is. Note that if you do an initial install (so 
extraConfig
is a dict), then later add an 
extraConfig
as a str (instead of a map), you'll get a warning that the old config couldn't be merged with the new config, but the new config is still applied. 
$ helm version
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}
I will update the docs to use a mapping example though so the warning isn't displayed, but things still work even with the warning.
s

Shaun Cutts

09/05/2020, 3:56 AM
@Jim Crist-Harif — is there perhaps a base image based on 
daskgateway/dask-gateway
which includes prefect, that I could/should use for further customization? With your speedy development, would be nice to have just one quickly improving dependency, rather than two. 🙂
j

Jim Crist-Harif

09/08/2020, 2:20 PM
Like JupyterHub, we (the dask-gateway "we", not the prefect "we") only provide a simple example base image, customization beyond that is left up to the user. If we start including images with assorted dependencies, we'd have a much larger support burden. I see you've added a PR to dask-gateway to make the base image more configurable, I'll hopefully get around to reviewing that tomorrow (lots to catch up on today).
#prefect-communityJoin Slack
Powered by