https://prefect.io
Join Slack
Cross Posting: <https://prefect-community.slack.com/archives/C048ZHT5U3U/p1681833200082759>
# ask-community
a
c
Hey Abhinav! We have been working on a new and (hopefully) improved way of setting this up with what we're calling "projects" - you can check out a tutorial here: https://docs.prefect.io/latest/tutorials/projects/#dockerized-deployment In short, you can create a Dockerfile, and then run 
prefect project init --recipe docker
to get things started. Note that the "docker" recipe will place your code within the docker filesystem. You can explore other available recipes (that include pushing / pulling code from remote locations such as S3) via 
prefect project recipe ls
a
I’ve looked into projects but I wanted to see if I could get the remote storage option working here with a base custom image. As I was mentioning in the thread from yesterday, I’d like to support the use case where multiple users are allowed to push independently. I could have them push a custom image but there’s no support for project within the SDK. Is there a way to get the SDK deployment option working where I specify a base image that is an extension of the prefect image with added dependencies?
It feels like it’s somehow path related but I can’t seem to figure out what’s going on.
I’ve traced it to this line:
RUN pip install --no-cache-dir --upgrade pip && pip install pip-tools && python -m piptools compile --resolver=backtracking pyproject.toml && pip install -r requirements.txt
Copy code
# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 prefecthq/prefect:2.10.3-python3.9

WORKDIR /install

COPY pyproject.toml .

RUN pip install --no-cache-dir --upgrade pip && pip install pip-tools && python -m piptools compile --resolver=backtracking pyproject.toml && pip install -r requirements.txt

WORKDIR /opt/prefect/flows
Without that line, eveyrthing works fine. I wonder if there’s a library issue between the prefect deps and my deps?
Sigh, this was a compatibility issue between adlfs and fsspec
c
Sorry just seeing this for some reason!
a
adlfs was 
2023.1.0
and fsspec was 
2023.4.0
pinning fsspec to 
2023.1.0
fixed the issue
c
Nice glad you figured it out! One day we will tackle improving Python environments…. 😅
a
haha
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by