Rajan Subramanian
03/23/2022, 6:45 PMAnna Geller
there must be some ability in the deployment spec to ensure a file thats not on a schedule gets run immediately upon deploymentDon't you think this would be rather undesirable? Imagine you create deployments from your CI/CD pipeline - you may not necessarily want that this triggers a run. Triggering flow runs is a different concern so it's better to keep it separate. You can always create flow runs via API or CLI easily after creating a deployment - LMK if you have any questions about that Regarding:
• i want the ability to constantly check if a task is running and if its not, to initiate a run. a signal or whatever.Can you create a separate thread for this question and clarify the problem you try to solve a bit more? In Prefect 2 you can build a while loop even though I'm not sure if having a never-ending flow would be helpful. You may prefer doing something like this to ensure your flow run ends at some time: https://gist.githubusercontent.com/anna-geller/b7bab79577e02d666f1c4befebdf3240/raw/61b71110a220289b3d8a22269ac91283dd79025e/03_flow_with_tasks.py
email on failureyou can do something like this: https://gist.github.com/53d4d86593658a6e63a4d317ad199b44 For the next time, it's much easier to post each question in a separate thread - it makes it easier to discuss if some answer requires clarification. It also makes it easier for us to transcribe the discussion into Discourse
Rajan Subramanian
04/05/2022, 10:45 PMaws ecr get-login-password --region <YOUR_AWS_REGION> | docker login --username AWS --password-stdin <YOUR_ECR_REGISTRY_ID>.dkr.ecr.<YOUR_AWS_REGION>.<http://amazonaws.com|amazonaws.com>
```Anna Geller
aws ecr create-repository --repository-name xxx
Rajan Subramanian
04/05/2022, 11:00 PM<YOUR_ECR_REGISTRY_ID>.dkr.ecr.<YOUR_AWS_REGION>.<http://amazonaws.com|amazonaws.com>
where am i getting the above value? sorry new to this ECRAnna Geller
prefecthq/prefect:latest
Rajan Subramanian
04/05/2022, 11:05 PMAnna Geller
Rajan Subramanian
04/05/2022, 11:07 PMAnna Geller
Rajan Subramanian
04/06/2022, 4:25 AMAnna Geller
Rajan Subramanian
04/06/2022, 1:11 PMAnna Geller
Rajan Subramanian
04/06/2022, 1:27 PMkubectl logs -lapp=orion --all-containers
and i keep getting an error "No work queue found named kubernetes" . So i then did
prefect work-queue create kubernetes
and I'm still getting that error. I'm using prefect cloud 2.0 for deployment.
prefect config view
PREFECT_PROFILE='default'
PREFECT_API_URL='https://api-beta.prefect.io/api/accounts/66153b97-4f15-4d43-8801-e4a0daf993cf/workspaces/b5cfa33d-78d2-43d2-b267-9747e409c78d'
PREFECT_API_KEY=secret keyAnna Geller
Rajan Subramanian
04/06/2022, 2:18 PM```import os
from os.path import join
from exchange_feeds.constants import EXCHANGEPATH, PHOBOSPATH
from prefect.deployments import DeploymentSpec
from prefect.flow_runners import KubernetesFlowRunner
path_to_pipeline = join(PHOBOSPATH, "pipelines", "feed_to_redis_pipeline.py")
path_to_file = os.path.join(EXCHANGEPATH, "feed_ingestor_to_redis.py")
DeploymentSpec(
name="ftx_blotter_btc-perp",
flow_location=path_to_pipeline,
tags=["ftx-blotter", "btc-perp"],
parameters={
"shell_task": path_to_file,
"symbol": "BTC-PERP",
"stream_name": "ftx-blotter",
},
flow_runner=KubernetesFlowRunner()
)
Anna Geller
flow_runner=KubernetesFlowRunner(
env=dict(
AWS_ACCESS_KEY_ID=os.environ.get("AWS_ACCESS_KEY_ID"),
AWS_SECRET_ACCESS_KEY=os.environ.get("AWS_SECRET_ACCESS_KEY"),
PREFECT_API_URL=os.environ.get("PREFECT_API_URL"),
PREFECT_API_KEY=os.environ.get("PREFECT_API_KEY"),
),
),
we have an open issue to make it easier, but it's still WIPRajan Subramanian
04/06/2022, 2:47 PM# syntax = docker/dockerfile:1.0-experimental
FROM python:3.9
# create a folder and cd into it
run mkdir temp_repo
run cd temp_repo
# set folder as current working directory
workdir /temp_repo
# move hell_world.py script from our local system to current workdir in docker
add hello_world.py .
# copy the requirements file from current system to docker directory
copy requirements.txt /temp_repo
run pip install -r requirements.txt
copy .gitconfig* /etc/gitconfig
copy ssh/config* /etc/ssh/ssh_config
copy ssh /root/.ssh
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan <http://github.com|github.com> >> ~/.ssh/known_hosts
RUN --mount=type=ssh pip install <git+ssh://git@github.com/phobos-capital/phobos-db.git>
RUN --mount=type=ssh git clone <git+ssh://git@github.com/phobos-capital/phobos-db.git>
RUN git clone phobos-trading-engine:phobos-capital/phobos-trading-engine.git
copy timescale.pem /root
run apt-get update -y
run apt-get upgrade -y
#run cd phobos-trading-engine
ENV PHOBOSSQLCONFIGPATH=./phobos-db/phobosdb/sql_config
ENV PHOBOSTRADINGENGINEPATH=./phobos-trading-engine
#switch to phobos-trading-engine path
run cd phobos-trading-engine
workdir /phobos-trading-engine
cd exchange_feeds
# login into prefect
prefect cloud login --key echo $PREFECT_API_KEY
Anna Geller
copy ssh /root/.ssh
Rajan Subramanian
04/06/2022, 2:51 PMAnna Geller
Rajan Subramanian
04/06/2022, 2:52 PMAnna Geller
Rajan Subramanian
04/06/2022, 2:53 PMdocker build --ssh default=$SSH_AUTH_SOCK .
Anna Geller
FROM prefecthq/prefect:2.0b2-python3.9
Rajan Subramanian
04/06/2022, 2:58 PMAnna Geller
Rajan Subramanian
04/06/2022, 2:58 PMERROR [20/20] RUN prefect cloud login --key $env_variable
when i remove the above statement, and enter into docker environment, im able to login to prefect cloud. i think the issue is with pressing enter when i try to login inside docker.Kevin Kho
prefect cloud login --key API_KEY --workspace kevinprefectio/prefect
Rajan Subramanian
04/06/2022, 5:14 PM