Rajan Subramanian
03/02/2022, 5:36 PMKevin Kho
Rajan Subramanian
03/02/2022, 9:53 PM@task(
name="run_streamer_in_shell",
description="runs each of the exchange.py file as a separate shell task",
retries=3,
retry_delay_seconds=10,
)
def run_streamer_in_shell(command: Task):
subprocess.run(command, shell=True)
@flow(name="live_feeds_to_redis_pipeline", task_runner=DaskTaskRunner())
def run_all_tasks(shell_tasks: List[str]):
for path in shell_tasks:
cmd = build_command(path)
run_streamer_in_shell(command=cmd)
and heres my deployment.py
STREAM_TASKS = ['path_to_ex1.py", "path_to_ex2.py, path_to_ex3.py"]
DeploymentSpec(
name="feed-to-redis-deployment",
_flow_location_="feed_to_redis_pipeline.py",
tags=["exchange-streamer", "raj"],
parameters={"shell_tasks": STREAM_TASKS},
)
``````Kevin Kho
Rajan Subramanian
03/02/2022, 9:56 PMKevin Kho
Rajan Subramanian
03/02/2022, 9:58 PMprefect orion database reset -y
Kevin Kho
Rajan Subramanian
03/02/2022, 10:08 PMKevin Kho
# Use 4 worker processes, each with 2 threads
DaskTaskRunner(
cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}
)
Rajan Subramanian
03/02/2022, 10:15 PMKevin Kho
Rajan Subramanian
03/02/2022, 10:23 PMKevin Kho
Rajan Subramanian
03/02/2022, 10:25 PMKevin Kho
Rajan Subramanian
03/03/2022, 9:49 PMKevin Kho
Rajan Subramanian
03/03/2022, 10:09 PMKevin Kho
Rajan Subramanian
03/03/2022, 10:12 PMKevin Kho
Rajan Subramanian
03/04/2022, 7:01 PMprefect deployment create ftx_deployment
however this isn't being picked up by the orion ui. doesn't the worker queue listen to all activate deployments if you dont specify a worker queue to a deployment?
initially, when i created deployments, i merely created a worker queue and did
prefect agent start <'uiud in string format
Kevin Kho
Rajan Subramanian
03/07/2022, 4:09 PMKevin Kho
Rajan Subramanian
03/09/2022, 6:24 PMKevin Kho
Rajan Subramanian
03/09/2022, 6:42 PMprefect create deployment.py
again where it contains the new deployment spec and previous deployment specsKevin Kho
Rajan Subramanian
03/09/2022, 6:45 PMprefect work-queue create feed-to-postgres-agent
prefect agent start 'uiud' obtained from previous step
I then go to the ui, go to deployments and hit quick run and see the runs as such:
I then go to my terminal and type:
prefect deployment create ftx/blotter_deployment.py
which creates 10 deployments from teh specs i have in that file.
i then go to my terminal and type
prefect deployment ls
and see the following:
live_feeds_to_redis_pipeline/binance-blotter for btcusdt
live_feeds_to_redis_pipeline/binance_blotter_ethusdt
live_feeds_to_redis_pipeline/binance_l1_btcusdt
live_feeds_to_redis_pipeline/binance_l1_ethusdt
live_feeds_to_redis_pipeline/ftx-blotter for avaxusd
live_feeds_to_redis_pipeline/ftx-blotter for bnbusd
live_feeds_to_redis_pipeline/ftx-blotter for btc-perp
live_feeds_to_redis_pipeline/ftx-blotter for btcusd
live_feeds_to_redis_pipeline/ftx-blotter for dogeusd
live_feeds_to_redis_pipeline/ftx-blotter for dotusd
live_feeds_to_redis_pipeline/ftx-blotter for ethusd
live_feeds_to_redis_pipeline/ftx-blotter for lunausd
live_feeds_to_redis_pipeline/ftx-blotter for solusd
live_feeds_to_redis_pipeline/ftx-blotter for xrpusd
redis_to_postgres_pipeline/redis-to-postgres-deployment
I refreshed my ui many times and dont see that being recognized. what can i do?Kevin Kho
Rajan Subramanian
03/09/2022, 10:06 PMKevin Kho
Rajan Subramanian
03/09/2022, 10:17 PMKevin Kho
Rajan Subramanian
03/09/2022, 10:26 PMKevin Kho
frn:1d
and fru:1d
at the top? Are you still trying to find the flows with scheduled or is this something else? Still waiting for the UI team on thatRajan Subramanian
03/09/2022, 10:40 PMlive_feeds_to_redis_pipeline/binance-blotter_btcusdt
live_feeds_to_redis_pipeline/binance_blotter_ethusdt
live_feeds_to_redis_pipeline/binance_l1_btcusdt
live_feeds_to_redis_pipeline/binance_l1_ethusdt
live_feeds_to_redis_pipeline/ftx_L1_avaxusd
live_feeds_to_redis_pipeline/ftx_L1_bnbusd
live_feeds_to_redis_pipeline/ftx_L1_btc-perp
live_feeds_to_redis_pipeline/ftx_L1_btcusd
live_feeds_to_redis_pipeline/ftx_L1_dogeusd
live_feeds_to_redis_pipeline/ftx_L1_dotusd
live_feeds_to_redis_pipeline/ftx_L1_ethusd
live_feeds_to_redis_pipeline/ftx_L1_lunausd
live_feeds_to_redis_pipeline/ftx_L1_solusd
live_feeds_to_redis_pipeline/ftx_L1_xrpusd
live_feeds_to_redis_pipeline/ftx_blotter_avaxusd
live_feeds_to_redis_pipeline/ftx_blotter_bnbusd
live_feeds_to_redis_pipeline/ftx_blotter_btc-perp
live_feeds_to_redis_pipeline/ftx_blotter_btcusd
live_feeds_to_redis_pipeline/ftx_blotter_dogeusd
live_feeds_to_redis_pipeline/ftx_blotter_dotusd
live_feeds_to_redis_pipeline/ftx_blotter_ethusd
live_feeds_to_redis_pipeline/ftx_blotter_lunausd
live_feeds_to_redis_pipeline/ftx_blotter_solusd
live_feeds_to_redis_pipeline/ftx_blotter_xrpusd
redis_to_postgres_pipeline/redis-to-postgres-deployment
Kevin Kho
Rajan Subramanian
03/10/2022, 12:50 AMKevin Kho
Rajan Subramanian
03/10/2022, 6:04 PMKevin Kho
Rajan Subramanian
03/10/2022, 6:10 PMKevin Kho
Rajan Subramanian
03/22/2022, 5:45 PMprefect deployment create deployment_name
if the agent hasn't started yet, its then scheduled to run once worker queue is created and run. thaksKevin Kho
Rajan Subramanian
03/22/2022, 5:52 PM