https://prefect.io logo
Channels
Powered by
r

Rajan Subramanian

03/23/2022, 6:45 PM
hi guys, when i hit run on a prefect ui on the cloud, its creating a separate run rather than just restarting the run i had just started. • is there a way to run it just once? • Another thing is, if i run a deployment file with many deployments, like 30, its kind of painful to manually go to the UI and hit run on all of them. there must be some ability in the deployment spec to ensure a file thats not on a schedule gets run immediately upon deployment. • i want the ability to constantly check if a task is running and if its not, to initiate a run. a signal or whatever. • ability to email my team and I in case something crashes on the prefect side that was not picked up on the linux side
a

Anna Geller

03/23/2022, 7:23 PM
Prefect 2 doesn't have restart functionality yet. In order for this to work, we will need to release the results first. You can follow the announcements channel to track the releases Regarding:
there must be some ability in the deployment spec to ensure a file thats not on a schedule gets run immediately upon deployment
Don't you think this would be rather undesirable? Imagine you create deployments from your CI/CD pipeline - you may not necessarily want that this triggers a run. Triggering flow runs is a different concern so it's better to keep it separate. You can always create flow runs via API or CLI easily after creating a deployment - LMK if you have any questions about that Regarding:
• i want the ability to constantly check if a task is running and if its not, to initiate a run. a signal or whatever.
Can you create a separate thread for this question and clarify the problem you try to solve a bit more? In Prefect 2 you can build a while loop even though I'm not sure if having a never-ending flow would be helpful. You may prefer doing something like this to ensure your flow run ends at some time: https://gist.githubusercontent.com/anna-geller/b7bab79577e02d666f1c4befebdf3240/raw/61b71110a220289b3d8a22269ac91283dd79025e/03_flow_with_tasks.py
email on failure
you can do something like this: https://gist.github.com/53d4d86593658a6e63a4d317ad199b44 For the next time, it's much easier to post each question in a separate thread - it makes it easier to discuss if some answer requires clarification. It also makes it easier for us to transcribe the discussion into Discourse
r

Rajan Subramanian

04/05/2022, 10:45 PM
hello, @Anna Geller, @Kevin Kho i was following your example about creating a eks fargate cluster. I see this command:
Copy code
aws ecr get-login-password --region <YOUR_AWS_REGION> | docker login --username AWS --password-stdin <YOUR_ECR_REGISTRY_ID>.dkr.ecr.<YOUR_AWS_REGION>.<http://amazonaws.com|amazonaws.com>
```
are you assuming that a ECR registry is already created on aws?
a

Anna Geller

04/05/2022, 10:53 PM
this command here is only to login. to create a repo, you can do:
Copy code
aws ecr create-repository --repository-name xxx
r

Rajan Subramanian

04/05/2022, 11:00 PM
@Anna Geller, sorry, what is this then?
Copy code
<YOUR_ECR_REGISTRY_ID>.dkr.ecr.<YOUR_AWS_REGION>.<http://amazonaws.com|amazonaws.com>
where am i getting the above value? sorry new to this ECR
a

Anna Geller

04/05/2022, 11:05 PM
this is the path to the ECR repo - AWS equivalent to e.g. 
prefecthq/prefect:latest
r

Rajan Subramanian

04/05/2022, 11:05 PM
oh
a

Anna Geller

04/05/2022, 11:06 PM
btw could you please not tag anyone? we will respond without tagging us, I promise 🙂 you can tag if there is some emergency
r

Rajan Subramanian

04/05/2022, 11:07 PM
oh ok sorry
a

Anna Geller

04/05/2022, 11:12 PM
no worries
r

Rajan Subramanian

04/06/2022, 4:25 AM
hi, in prefect orion, how does one set the aws kubernetes cluster as the execution layer? Based on the example here, https://towardsdatascience.com/distributed-data-pipelines-made-easy-with-aws-eks-and-prefect-106984923b30, i was trying to follow along but its hard since prefect 2 is different than prefect 1. Im stuck in the part about adding storage information and pushing flow to ECR. is there a way to dockerize the flow and push the image to ECR?
i am trying to deploy prefect 2.0 using eks/fargate
a

Anna Geller

04/06/2022, 10:05 AM
This article is super outdated. In general, there is no Docker storage in Orion, but you can package your dependencies into Docker image and push it to ECR. But flow storage would still be S3.
r

Rajan Subramanian

04/06/2022, 1:11 PM
oh ok, so would DockerFlowRunner() be used here? Or can i just use my regular flowrunner and package my dependencies in a docker container prior to fargate execution?
a

Anna Geller

04/06/2022, 1:17 PM
can you summarize your setup?
So far, I understood: you run Orion on an EC2 instance. You then want to package your flow dependencies into Docker images and push those to ECR. If you use DockerFlowRunner and reference the ECR image on this flow runner, you are good to go, and everything will be executed on this EC2 instance. But we don't have ECS Fargate support in 2.0 yet. If you want to use Fargate, the only option would be using EKS on Fargate (Kubernetes on Fargate) and a KubernetesFlowRunner (rather than DockerFlowRunner)
r

Rajan Subramanian

04/06/2022, 1:27 PM
oh i dropped EC2. im using EKS now on AWS with FARGATE option. Ok then kubernetesFlowRunner it is. thanks! I was thinking about your suggestions yesterday. I just wanted to get to a point where i was comfortable testing it locally. But your suggestion to use FARGATE is much better. Also in kubernetesFlowRunner(), I tried,
Copy code
kubectl logs -lapp=orion --all-containers
and i keep getting an error "No work queue found named kubernetes" . So i then did
Copy code
prefect work-queue create kubernetes
and I'm still getting that error. I'm using prefect cloud 2.0 for deployment.
Copy code
prefect config view
PREFECT_PROFILE='default' PREFECT_API_URL='https://api-beta.prefect.io/api/accounts/66153b97-4f15-4d43-8801-e4a0daf993cf/workspaces/b5cfa33d-78d2-43d2-b267-9747e409c78d' PREFECT_API_KEY=secret key
a

Anna Geller

04/06/2022, 2:17 PM
can you share your DeploymentSpec? (redact any sensitive info)
r

Rajan Subramanian

04/06/2022, 2:18 PM
Copy code
```import os
from os.path import join

from exchange_feeds.constants import EXCHANGEPATH, PHOBOSPATH
from prefect.deployments import DeploymentSpec
from prefect.flow_runners import KubernetesFlowRunner

path_to_pipeline = join(PHOBOSPATH, "pipelines", "feed_to_redis_pipeline.py")
path_to_file = os.path.join(EXCHANGEPATH, "feed_ingestor_to_redis.py")


DeploymentSpec(
    name="ftx_blotter_btc-perp",
    flow_location=path_to_pipeline,
    tags=["ftx-blotter", "btc-perp"],
    parameters={
        "shell_task": path_to_file,
        "symbol": "BTC-PERP",
        "stream_name": "ftx-blotter",
    },
    flow_runner=KubernetesFlowRunner()
)
a

Anna Geller

04/06/2022, 2:20 PM
thanks! I think you need to add some env variables to your flow runner to make it work with EKS and S3 storage:
Copy code
flow_runner=KubernetesFlowRunner(
        env=dict(
            AWS_ACCESS_KEY_ID=os.environ.get("AWS_ACCESS_KEY_ID"),
            AWS_SECRET_ACCESS_KEY=os.environ.get("AWS_SECRET_ACCESS_KEY"),
            PREFECT_API_URL=os.environ.get("PREFECT_API_URL"),
            PREFECT_API_KEY=os.environ.get("PREFECT_API_KEY"),
        ),
    ),
we have an open issue to make it easier, but it's still WIP
to be clear, those don't have to be env variables, you can hardcode them if you like 😄 my point here is only that atm you need to set the API_KEY and the API_URL on the flow runner to make it work with a remote infra like EKS on Fargate
😆 1
not sure if AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are needed f you attach an IAM role with S3 permissions to your EKS cluster
r

Rajan Subramanian

04/06/2022, 2:47 PM
oh i see thanks, yea im passing those as env variables into my docker image by loading it from a docker file. Just to be clear, I'm assuming I still have to create a docker image even though I'm using a kubernetees flow runner correct? Since i need to upload those images to ECR
At the moment, my docker file looks something like this:
Copy code
# syntax = docker/dockerfile:1.0-experimental
FROM python:3.9

# create a folder and cd into it
run mkdir temp_repo
run cd temp_repo

# set folder as current working directory
workdir /temp_repo

# move hell_world.py script from our local system to current workdir in docker
add hello_world.py .

# copy the requirements file from current system to docker directory
copy requirements.txt /temp_repo
run pip install -r requirements.txt


copy .gitconfig* /etc/gitconfig
copy ssh/config* /etc/ssh/ssh_config
copy ssh /root/.ssh
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan <http://github.com|github.com> >> ~/.ssh/known_hosts

RUN --mount=type=ssh pip install <git+ssh://git@github.com/phobos-capital/phobos-db.git>
RUN --mount=type=ssh git clone <git+ssh://git@github.com/phobos-capital/phobos-db.git>
RUN git clone phobos-trading-engine:phobos-capital/phobos-trading-engine.git

copy timescale.pem /root

run apt-get update -y
run apt-get upgrade -y
#run cd phobos-trading-engine

ENV PHOBOSSQLCONFIGPATH=./phobos-db/phobosdb/sql_config
ENV PHOBOSTRADINGENGINEPATH=./phobos-trading-engine

#switch to phobos-trading-engine path

run cd phobos-trading-engine
workdir /phobos-trading-engine
cd exchange_feeds
# login into prefect
prefect cloud login --key echo $PREFECT_API_KEY
a

Anna Geller

04/06/2022, 2:51 PM
just to flag it: this could be risky from security perspective 
copy ssh /root/.ssh
r

Rajan Subramanian

04/06/2022, 2:51 PM
its literally impossible to copy multiple private repositories in docker
a

Anna Geller

04/06/2022, 2:51 PM
why would you want to do this? can you explain the problem you try to solve this way?
r

Rajan Subramanian

04/06/2022, 2:52 PM
Do what copy ssh /root/.ssh?
a

Anna Geller

04/06/2022, 2:52 PM
having SSH keys baked in your container
r

Rajan Subramanian

04/06/2022, 2:53 PM
ahh...because i was having trouble cloning and doing an pip install of private repositories
Copy code
docker build --ssh default=$SSH_AUTH_SOCK .
this only works when u want to pip install one private repository, doesn't work for multiple
if you can recommend a way to pip install multiple private repo securely im happy to make changes
ah i want to do this because A) i have a private repo thats a postgres database reader/writer using async methods. i also have another private repo where i am doing all the exchange streaming of raw data called phobos-trading-engine repo that has prefect deployments speced out. Hence this method.
a

Anna Geller

04/06/2022, 2:57 PM
I'm no security expert. I just wanted to make sure you are aware of the trade-offs here. I'm trying to avoid SSH keys and leverage e.g. GitHub personal access tokens when possible. Back to the original question: I think you can actually start with Prefect base image instead of Python, which will make things way easier:
Copy code
FROM prefecthq/prefect:2.0b2-python3.9
r

Rajan Subramanian

04/06/2022, 2:58 PM
yep that was the next step 🙂 im learning docker and kubernetees in a day or so. so this is my hackey way of doing things.
👍 1
a

Anna Geller

04/06/2022, 2:58 PM
perhaps it's just easier to put it all into the same repo so that you don't need to clone the repos back and forth? just mentioning this as an option 😄
r

Rajan Subramanian

04/06/2022, 2:58 PM
yea that was another option. i think we might resort to that
🙌 1
Hello, how do i login to prefect cloud in docker? Im getting an error:
Copy code
ERROR [20/20] RUN prefect cloud login --key $env_variable
when i remove the above statement, and enter into docker environment, im able to login to prefect cloud. i think the issue is with pressing enter when i try to login inside docker.
k

Kevin Kho

04/06/2022, 5:09 PM
Try adding a -y. Will test now
I was wrong. Just specify your workspace already and you won’t need to press enter
Copy code
prefect cloud login --key API_KEY --workspace kevinprefectio/prefect
r

Rajan Subramanian

04/06/2022, 5:14 PM
oh awesome let me try
21 Views
Powered by