Cross Posting: <https://prefect-community.slack.co...
# ask-communitya
Abhinav Chordia
04/18/2023, 5:50 PMc
Chris White
04/18/2023, 5:52 PM
Hey Abhinav! We have been working on a new and (hopefully) improved way of setting this up with what we're calling "projects" - you can check out a tutorial here: https://docs.prefect.io/latest/tutorials/projects/#dockerized-deployment
In short, you can create a Dockerfile, and then run
prefect project init --recipe dockerprefect project recipe lsa
Abhinav Chordia
04/18/2023, 5:57 PMI’ve looked into projects but I wanted to see if I could get the remote storage option working here with a base custom image. As I was mentioning in the thread from yesterday, I’d like to support the use case where multiple users are allowed to push independently. I could have them push a custom image but there’s no support for project within the SDK.
Is there a way to get the SDK deployment option working where I specify a base image that is an extension of the prefect image with added dependencies?
Abhinav Chordia
04/18/2023, 5:58 PMIt feels like it’s somehow path related but I can’t seem to figure out what’s going on.
Abhinav Chordia
04/18/2023, 7:21 PMI’ve traced it to this line:
RUN pip install --no-cache-dir --upgrade pip && pip install pip-tools && python -m piptools compile --resolver=backtracking pyproject.toml && pip install -r requirements.txt
Abhinav Chordia
04/18/2023, 7:21 PM Copy code
# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 prefecthq/prefect:2.10.3-python3.9
WORKDIR /install
COPY pyproject.toml .
RUN pip install --no-cache-dir --upgrade pip && pip install pip-tools && python -m piptools compile --resolver=backtracking pyproject.toml && pip install -r requirements.txt
WORKDIR /opt/prefect/flowsAbhinav Chordia
04/18/2023, 7:22 PMWithout that line, eveyrthing works fine. I wonder if there’s a library issue between the prefect deps and my deps?
Abhinav Chordia
04/18/2023, 11:41 PMSigh, this was a compatibility issue between adlfs and fsspec
c
Chris White
04/18/2023, 11:41 PM
Sorry just seeing this for some reason!
a
Abhinav Chordia
04/18/2023, 11:41 PMadlfs was
2023.1.02023.4.0Abhinav Chordia
04/18/2023, 11:41 PMpinning fsspec to
2023.1.0c
Chris White
04/18/2023, 11:42 PM
Nice glad you figured it out! One day we will tackle improving Python environments…. 😅
a
Abhinav Chordia
04/18/2023, 11:42 PMhaha
2 Views