https://prefect.io logo
Join Slack
Powered by
Hi all, I'm trying to debug custom docker images. ...
# prefect-server
m
Hi all, I'm trying to debug custom docker images. My reference setup is
Copy code
# Dockerfile
FROM prefecthq/prefect:0.13.5-python3.8
Copy code
docker build -t prefect-base .
Everything is working as expected when using (i) the local docker-compose server and UI, (ii) a minimal flow
Copy code
# register.py

import prefect
from prefect import task, Flow
from prefect.environments.storage import Docker


@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("test message")

flow = Flow("prefect-base", tasks=[hello_task])
base_image = "prefect-base:latest"

flow.storage = Docker(base_image=base_image, local_image=True)
flow.register(project_name="test")
and (iii) a docker agent running via
Copy code
prefect agent start docker --no-pull -v
However, when I change to:
Copy code
prefect agent start docker --no-pull -v --show-flow-logs
I'm getting an error (upon triggering a run in the UI) of the form:
Copy code
[2020-09-09 13:21:44,229] DEBUG - agent | Updating states for flow run 457ba6ff-9a8e-4117-b563-166720de3ba9
[2020-09-09 13:21:44,230] DEBUG - agent | Flow run 457ba6ff-9a8e-4117-b563-166720de3ba9 is in a Scheduled state, updating to Submitted
[2020-09-09 13:21:44,262] INFO - agent | Deploying flow run 457ba6ff-9a8e-4117-b563-166720de3ba9
[2020-09-09 13:21:44,263] DEBUG - agent | Creating Docker container prefect-base:2020-09-09t13-21-12-637383-00-00
[2020-09-09 13:21:44,315] DEBUG - agent | Starting Docker container with ID 4e2404b2148c82b43ec9185d2c553ec1dc8b6ad8a653267c54664f7eae28d77c
[2020-09-09 13:21:44,483] DEBUG - agent | Querying for flow runs
[2020-09-09 13:21:44,501] DEBUG - agent | No flow runs found
[2020-09-09 13:21:44,501] DEBUG - agent | Next query for flow runs in 0.5 seconds
[2020-09-09 13:21:44,523] ERROR - agent | Logging platform error for flow run 457ba6ff-9a8e-4117-b563-166720de3ba9
[2020-09-09 13:21:44,568] ERROR - agent | Error while deploying flow: AttributeError("Can't pickle local object 'Agent.run_heartbeat_thread.<locals>.run'")
[2020-09-09 13:21:44,568] DEBUG - agent | Completed flow run submission (id: 457ba6ff-9a8e-4117-b563-166720de3ba9)
My main aim is to determine why I can get the flow above to run with the following custom Dockerfile:
Copy code
FROM python:3.8-slim

# Set system locale
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8

RUN apt update && \
    apt install -y gcc git && \
    mkdir /root/.prefect/ && \
    pip install --no-cache-dir prefect[kubernetes] && \
    apt remove -y git && \
    apt clean && apt autoremove -y && \
    rm -rf /var/lib/apt/lists/*
but not with this Dockerfile:
Copy code
FROM ubuntu:20.04

# Set system locale
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8

RUN apt update && \
    apt install -y gcc git python3-pip && \
    mkdir /root/.prefect/ && \
    ln -s /usr/bin/pip3 /usr/bin/pip && \
    ln -s /usr/bin/python3 /usr/bin/python && \
    pip install --no-cache-dir prefect[kubernetes] && \
    apt remove -y git && \
    apt clean && apt autoremove -y && \
    rm -rf /var/lib/apt/lists/*
j
Hmm that’s weird, I’m not sure why it would be attempting to pickle the agent’s heartbeat thread. Judging by the fact that you are able to run successfully with a different base image configuration I’m guessing it might be something related with the dependencies in that image (mainly the 
RUN
statement in the Dockerfile). I don’t have an immediate answer but would you mind transitioning this over to an issue on the prefect repo with a reproducible example and this information you provided?
m
Okay. I can make a github gist with the inputs and outputs.
j
That would be great!
m
Hi @josh, I've opened an issue here https://github.com/PrefectHQ/prefect/issues/3288
👍 1
@Marvin
m
This contest is the sort of thing you lifeforms enjoy, is it? I've entered you to win, @masonlr.
2 Views
Open in Slack
PreviousNext
Powered by