Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Abhinav Chordia
04/18/2023, 5:50 PMc
Chris White
04/18/2023, 5:52 PMHey Abhinav! We have been working on a new and (hopefully) improved way of setting this up with what we're calling "projects" - you can check out a tutorial here: https://docs.prefect.io/latest/tutorials/projects/#dockerized-deployment
In short, you can create a Dockerfile, and then run
prefect project init --recipe dockerprefect project recipe lsa
Abhinav Chordia
04/18/2023, 5:57 PMI’ve looked into projects but I wanted to see if I could get the remote storage option working here with a base custom image. As I was mentioning in the thread from yesterday, I’d like to support the use case where multiple users are allowed to push independently. I could have them push a custom image but there’s no support for project within the SDK.
Is there a way to get the SDK deployment option working where I specify a base image that is an extension of the prefect image with added dependencies?
It feels like it’s somehow path related but I can’t seem to figure out what’s going on.
I’ve traced it to this line:
RUN pip install --no-cache-dir --upgrade pip && pip install pip-tools && python -m piptools compile --resolver=backtracking pyproject.toml && pip install -r requirements.txt
# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 prefecthq/prefect:2.10.3-python3.9
WORKDIR /install
COPY pyproject.toml .
RUN pip install --no-cache-dir --upgrade pip && pip install pip-tools && python -m piptools compile --resolver=backtracking pyproject.toml && pip install -r requirements.txt
WORKDIR /opt/prefect/flowsWithout that line, eveyrthing works fine. I wonder if there’s a library issue between the prefect deps and my deps?
Sigh, this was a compatibility issue between adlfs and fsspec
c
Chris White
04/18/2023, 11:41 PMSorry just seeing this for some reason!
a
Abhinav Chordia
04/18/2023, 11:41 PMadlfs was
2023.1.02023.4.0pinning fsspec to
2023.1.0c
Chris White
04/18/2023, 11:42 PMNice glad you figured it out! One day we will tackle improving Python environments…. 😅
a
Abhinav Chordia
04/18/2023, 11:42 PMhaha