https://prefect.io
Join Slack
Hey all, I'm attempting to switch from a local agent on a VM to a docker agent. When using docker-co...
# ask-community
j
Hey all, I'm attempting to switch from a local agent on a VM to a docker agent. When using docker-compose with image: prefecthq/prefect:latest I'm getting the following error. Does anyone have a clue what's going on here? Thanks!
a
Seems like urllib3 issue. Can you try the Prefect version from master and see if this fixes your issue?
👀 1
j
Hi Anna, thanks for the quick response. I am using prefect==0.15.10, which should be the newest version
This is currently the Dockerfile I'm trying with
a
@Jelle Vegter correct, but this PR was added after 0.15.10 release
if you could try the version from the master branch, it would be great because it contains a fix to an urllib3 issue
Ignore what I said about urllib3, I think you are facing a bigger problem: the Docker agent process cannot run as a Docker container, it should run as a local process that deploys flow runs as containers. Otherwise you’ll end up running your flow runs within the agent’s Docker container rather than as independent containers.
👀 1
j
ah okay. So if I understand correctly, I need to build the docker image with all the dependencies but run the agent start command locally.
a
Correct. Can you try creating a virtual environment with the same package dependencies, and run your “prefect agent docker start” command in a virtual environment? This should fix your problem.
Additionally, the --key option doesn’t exist for the Docker agent. To authenticate your agent with the API key, you would need to authenticate your terminal first:
Copy code
prefect auth login ‒-key XXX
and then start the agent:
Copy code
prefect agent docker start --label YOUR_AGENT_LABEL
it would be great if you assign the label explicitly to your agent to avoid any issues down the road
j
Amazing, thanks. I'll attempt it right now
👍 1
Hi Anna, another question if you don't mind. I developed the flow with Python 3.8.8, which is why my flows (run by the Docker agent) are failing. I'm trying to understand how to sync Python version.
I schanged my dockerfile to use a non-Master image: prefecthq/prefect:latest-python3.8 so that the Python version is the same, but I am still getting the Python missmatch error.
Is the container image which runs the flows not determined by this Dockerfile? Which means the Prefect forces 3.7.12 and that I need to start building flows with this Python version?
a
@Jelle Vegter can you run “prefect diagnostics” and share your output?
j
k
There are three versions going on here. The registration version, agent version, and the image version. That message is from the agent unpickling the registered flow so it’s the agent version and registration version that have mismatches. I think ideally the image version for DockerRun should also be the same
a
Just to add to Kevin’s answer (since I was typing at the same time anyway 😄) the Prefect version of the environment from which your register your flow and the execution environment (agent) should match. When you use Docker agent but the flow is serialized, say with Prefect 3.7.12, then when you run it in 3.8.8 environment, it may cause issues during deserialization in a different Python environment. So you can either: 1. Try to match the registration environment to what is used in the execution layer (agent) 2. Switch to 
stored_as_script=True
in your Docker storage. Do you even use Docker storage or Azure storage + DockerRun with your custom image? Sharing your updated Dockerfile and your storage + run config would help
j
I'm fighting with a few Python install issues atm 😉. I'll post as soon as I fixed them
Really appreciate the help guys!
🙌 1
Is there a way to change what Python is on the agent? I installed Python 3.7 on my dev VM but it seems to have issues installing prefect[azure] on Azure ml components
Oh, I am reading online that the 3.7 version is caused by the pickled storage format. I will try to switch from the current storage to a docker storage
👍 1
k
The agent will use the Python installation you run it from. If you have anaconda, or some other venv manager, you can start create a new environment with the specified python version and then start the agent from that.
From experience this prefect[azure] installs those azure tools and they have so many version conflicts
r
To follow up on Jelle's questions, when we have a local docker agent running on our VM, a new container is created where the flow is to be executed. Is it possible to create these containers based on a local docker image or does the template-dockerimage always needs to be downloaded from an online location? The error we are facing is that the prefecthq/prefect:latest does not contain pandas. We would like to use our custom image for flexibility in dependencies.
our issue is that we seem to create a docker image which is not used when flows are executed. Everytime a flow is executed a new container based on a different image is created
a
@Ruben Sik you can definitely use your own custom image. You would need to push it to some container registry so that prefect can find and pull it. Alternatively, if you want to use only local images, you can leverage Docker storage and set registry_url to None:
Copy code
with Flow(
    "example-docker-storage",
    storage=Docker(registry_url=None, image_name="your_image", image_tag="your_tag"),
) as flow:
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by