Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
m
Max Lei
03/01/2022, 7:23 PMI'm trying to test
DaskExecutorDockerRuna
Anna Geller
03/01/2022, 7:30 PMActually, the easiest way to test DaskExecutor would be to run the workflow locally without Docker first. But it depends on what do you try to do. We've recently updated this topic that shows how you can use LocalDaskExecutor and how/when to move to DaskExecutor - perhaps is this what you are looking for? https://discourse.prefect.io/t/what-is-the-difference-between-a-daskexecutor-and-a-localdaskexecutor/374
But if you ask how you can pass a Docker image to your Dask cluster class, you could provide that through a cluster_kwargs like so:
flow.executor = DaskExecutor(
# FargateCluster is just one example
cluster_class="dask_cloudprovider.aws.FargateCluster",
cluster_kwargs={"n_workers": 4, "image": "your-prefect-image"},
)m
Max Lei
03/01/2022, 7:36 PM@Anna Geller I had the flow working on the LocalDaskExecutor, but I'm trying to make sure the Docker image I have works. It's kind of cumbersome to test it directly on a Fargate cluster, so that's why I'm trying to test it locally first.
a
Anna Geller
03/01/2022, 7:42 PMgotcha, have you tried coiled? they make it so much easier to test various cluster configuration and build the proper containerized environment for Dask
@Max Lei alternatively, if you have a local Kubernetes cluster on your machine, you could test it that way perhaps?
with Flow(
"Dask Kubernetes Flow",
storage=storage,
executor=DaskExecutor(
cluster_class=lambda: KubeCluster(make_pod_spec(image=prefect.context.image)),
adapt_kwargs={"minimum": 2, "maximum": 3},
),
run_config=KubernetesRun(),
) as flow:k
Kevin Kho
03/01/2022, 7:52 PMI think testing locally is still not enough to guarantee it works on Fargate because of IAM roles and image compatibility and stuff like that
m
Max Lei
03/01/2022, 7:53 PM@Anna Geller Unfortunately I have no idea how to use kubernetes. Would
DockerRun:upvote: 1
@Kevin Kho I have a fargate cluster with the correctly configured roles, tested it with docker images deploys using the local CLI etc as close as what prefect would do.
k
Kevin Kho
03/01/2022, 7:55 PMI would say DockerRun is the closest option yep unless Anna says otherwise
a
Anna Geller
03/01/2022, 7:57 PMwhat do you mean by Fargate instance? Fargate is serverless so there are no instances 🙃 or do you mean you have some ECS Fargate service running 24/7 for testing?
I just can't grasp what would you test with just DockerRun because it wouldn't run on a cluster but just in a container right? but it of course depends on what you try to do
maybe also just building your image, then doing:
docker run --rm -it image_namek
Kevin Kho
03/01/2022, 8:00 PMI think Max is thinking that if the Flow runs with DockerRun successfully, he knows all the dependencies are there and then if he uses the Docker image for the Dask cluster, it should work as well. I think it makes sense
👍 1
a
Anna Geller
03/01/2022, 8:01 PMmy poor misuse of the matrix 😅 https://imgflip.com/i/6721ma
😅 1
m
Max Lei
03/01/2022, 8:03 PM@Anna Geller Yeah I mean an ECS Fargate service. I'm pretty sure the service works, or at least I know how to fix it if the issue are permission/images.
I'm looking at your repo: https://github.com/anna-geller/packaging-prefect-flows/blob/master/Dockerfile#L3, I placed my flows in /opt/prefect/flows, but I'm getting
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/ubuntu/'")a
Anna Geller
03/01/2022, 8:06 PMI believe it's a user issue within your container. Can you share your Dockerfile?
k
Kevin Kho
03/01/2022, 8:06 PMI think this is a Storage issue where Local storage was used?
m
Max Lei
03/01/2022, 8:06 PMFROM prefecthq/prefect:0.11.4-python3.7
COPY ./dist/<PACKAGE_NAME_REMOVED>-0.0.0.tar.gz .
COPY requirements.txt .
COPY gh ./.config/
RUN apt update
RUN apt install -y curl
RUN curl -fsSL <https://cli.github.com/packages/githubcli-archive-keyring.gpg> | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg && \
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] <https://cli.github.com/packages> stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null
RUN apt update
RUN apt install -y build-essential gcc gh
RUN pip install setuptools==50.0.0
RUN pip install pip==20.2
RUN pip install pystan==2.17.1.0
RUN pip install fbprophet==0.7.1
RUN pip install -r requirements.txt
RUN pip install <PACKAGE_NAME_REMOVED>-0.0.0.tar.gz
WORKDIR /opt/prefect
COPY . /opt/prefect/flows/I did not setup a storage, I'm assuming it's the default local storage?
Sounds like you need an S3 storage for this and also
DaskExecutora
Anna Geller
03/01/2022, 8:08 PMyup Kevin, that could be when using Docker storage.
@Max Lei would you consider upgrading to a more recent version than 0.11.4? a loooot has changed since then. We have 1.0.0 now 😎
k
Kevin Kho
03/01/2022, 8:09 PMYou need any remote storage that the agent can pull Github/S3/etc. It is just looking for the path locally
:upvote: 1
a
Anna Geller
03/01/2022, 8:09 PMthx for sharing the Dockerfile, can you also share your Flow definition, especially the storage and run config? I could try to reproduce or investigate
m
Max Lei
03/01/2022, 8:10 PMThat part I can't share, but I only have
DockerRun(image="<IMAGE_NAME_REMOVED>:latest")k
Kevin Kho
03/01/2022, 8:11 PMYeah you’ll probably just need to add any remote Storage and make sure your agent is authenticated to pull from it
fbprophetm
Max Lei
03/01/2022, 8:12 PM@Kevin Kho I'm sort of confused why the agent would need a remote storage though? Isn't everything required for the agent local?
Yeah it is, if
fbprophetI've heard GreyKite provides similar functionality and graphing but without the install headache.
k
Kevin Kho
03/01/2022, 8:17 PMWhen the agent polls Prefect for flows to run, it will find a Flow and then load in the configuration from the database (Storage, RunConfig). It then looks for the storage, fetches the flow definition from there, and runs it on top of the RunConfig.
It looks like your registration machine is saving the Flow locally (default Local storage) and then the agent is on a different machine that looks for that file path but it doesn’t exist.
So you need storage to be something like S3/Github where the agent can find it and pull it down and then execute it. Does that make sense?
Greykite…my current impression is it’s just a toy for now?
a
Anna Geller
03/01/2022, 8:17 PMPrefect deploys a Docker container for your flow run and then within that Docker container it pulls your flow from Storage. That's why you need remote storage, especially later when you use Fargate
k
Kevin Kho
03/01/2022, 8:18 PMOr yes what Anna said use Docker Storage to find the Flow file inside the container that you already have
👍 1
a
Anna Geller
03/01/2022, 8:19 PMand if you really want to use local storage with Docker, you can try this https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/local_script_docker_run_local_image.py
m
Max Lei
03/01/2022, 8:20 PMAh ok, I see, I'll try setting up a storage and retry.
👍 1
@Kevin Kho Greykite is actually not that bad, performs pretty similar to fbprophet when you need rough forecasts.
👍 1
m
Matthias
03/01/2022, 8:44 PMAlso, @Max Lei, looking at you dockerfile I noticed there are a lot of RUN cmd’s in there. Note that every cmd creates a layer in your image and that more layers means a larger sized image. You can slim down by combining related cmd’s into one layer…
:upvote: 3