Rajan Subramanian
03/02/2022, 5:36 PMKevin Kho
Rajan Subramanian
03/02/2022, 9:53 PM@task(
name="run_streamer_in_shell",
description="runs each of the exchange.py file as a separate shell task",
retries=3,
retry_delay_seconds=10,
)
def run_streamer_in_shell(command: Task):
subprocess.run(command, shell=True)
@flow(name="live_feeds_to_redis_pipeline", task_runner=DaskTaskRunner())
def run_all_tasks(shell_tasks: List[str]):
for path in shell_tasks:
cmd = build_command(path)
run_streamer_in_shell(command=cmd)
and heres my deployment.py
STREAM_TASKS = ['path_to_ex1.py", "path_to_ex2.py, path_to_ex3.py"]
DeploymentSpec(
name="feed-to-redis-deployment",
_flow_location_="feed_to_redis_pipeline.py",
tags=["exchange-streamer", "raj"],
parameters={"shell_tasks": STREAM_TASKS},
)
``````Rajan Subramanian
03/02/2022, 9:54 PMKevin Kho
Kevin Kho
Rajan Subramanian
03/02/2022, 9:56 PMKevin Kho
Rajan Subramanian
03/02/2022, 9:58 PMprefect orion database reset -y
Rajan Subramanian
03/02/2022, 9:59 PMKevin Kho
Rajan Subramanian
03/02/2022, 10:08 PMKevin Kho
Kevin Kho
# Use 4 worker processes, each with 2 threads
DaskTaskRunner(
cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}
)
Rajan Subramanian
03/02/2022, 10:15 PMRajan Subramanian
03/02/2022, 10:15 PMKevin Kho
Rajan Subramanian
03/02/2022, 10:23 PMRajan Subramanian
03/02/2022, 10:23 PMKevin Kho
Rajan Subramanian
03/02/2022, 10:25 PMKevin Kho
Rajan Subramanian
03/03/2022, 9:49 PMKevin Kho
Rajan Subramanian
03/03/2022, 10:09 PMRajan Subramanian
03/03/2022, 10:10 PMKevin Kho
Rajan Subramanian
03/03/2022, 10:12 PMRajan Subramanian
03/04/2022, 5:36 PMKevin Kho
Rajan Subramanian
03/04/2022, 7:01 PMRajan Subramanian
03/07/2022, 3:59 PMprefect deployment create ftx_deployment
however this isn't being picked up by the orion ui. doesn't the worker queue listen to all activate deployments if you dont specify a worker queue to a deployment?
initially, when i created deployments, i merely created a worker queue and did
prefect agent start <'uiud in string format
Rajan Subramanian
03/07/2022, 4:00 PMKevin Kho
Rajan Subramanian
03/07/2022, 4:09 PMRajan Subramanian
03/07/2022, 4:10 PMKevin Kho
Kevin Kho
Rajan Subramanian
03/09/2022, 6:24 PMKevin Kho
Rajan Subramanian
03/09/2022, 6:42 PMprefect create deployment.py
again where it contains the new deployment spec and previous deployment specsKevin Kho
Rajan Subramanian
03/09/2022, 6:45 PMRajan Subramanian
03/09/2022, 8:43 PMprefect work-queue create feed-to-postgres-agent
prefect agent start 'uiud' obtained from previous step
I then go to the ui, go to deployments and hit quick run and see the runs as such:
I then go to my terminal and type:
prefect deployment create ftx/blotter_deployment.py
which creates 10 deployments from teh specs i have in that file.
i then go to my terminal and type
prefect deployment ls
and see the following:
live_feeds_to_redis_pipeline/binance-blotter for btcusdt
live_feeds_to_redis_pipeline/binance_blotter_ethusdt
live_feeds_to_redis_pipeline/binance_l1_btcusdt
live_feeds_to_redis_pipeline/binance_l1_ethusdt
live_feeds_to_redis_pipeline/ftx-blotter for avaxusd
live_feeds_to_redis_pipeline/ftx-blotter for bnbusd
live_feeds_to_redis_pipeline/ftx-blotter for btc-perp
live_feeds_to_redis_pipeline/ftx-blotter for btcusd
live_feeds_to_redis_pipeline/ftx-blotter for dogeusd
live_feeds_to_redis_pipeline/ftx-blotter for dotusd
live_feeds_to_redis_pipeline/ftx-blotter for ethusd
live_feeds_to_redis_pipeline/ftx-blotter for lunausd
live_feeds_to_redis_pipeline/ftx-blotter for solusd
live_feeds_to_redis_pipeline/ftx-blotter for xrpusd
redis_to_postgres_pipeline/redis-to-postgres-deployment
I refreshed my ui many times and dont see that being recognized. what can i do?Kevin Kho
Kevin Kho
Kevin Kho
Rajan Subramanian
03/09/2022, 10:06 PMRajan Subramanian
03/09/2022, 10:06 PMRajan Subramanian
03/09/2022, 10:07 PMKevin Kho
Rajan Subramanian
03/09/2022, 10:17 PMRajan Subramanian
03/09/2022, 10:18 PMRajan Subramanian
03/09/2022, 10:18 PMRajan Subramanian
03/09/2022, 10:19 PMRajan Subramanian
03/09/2022, 10:19 PMRajan Subramanian
03/09/2022, 10:20 PMKevin Kho
Rajan Subramanian
03/09/2022, 10:26 PMKevin Kho
frn:1d
and fru:1d
at the top? Are you still trying to find the flows with scheduled or is this something else? Still waiting for the UI team on thatKevin Kho
Kevin Kho
Rajan Subramanian
03/09/2022, 10:40 PMRajan Subramanian
03/09/2022, 10:40 PMRajan Subramanian
03/09/2022, 11:44 PMRajan Subramanian
03/09/2022, 11:48 PMRajan Subramanian
03/09/2022, 11:50 PMRajan Subramanian
03/09/2022, 11:50 PMlive_feeds_to_redis_pipeline/binance-blotter_btcusdt
live_feeds_to_redis_pipeline/binance_blotter_ethusdt
live_feeds_to_redis_pipeline/binance_l1_btcusdt
live_feeds_to_redis_pipeline/binance_l1_ethusdt
live_feeds_to_redis_pipeline/ftx_L1_avaxusd
live_feeds_to_redis_pipeline/ftx_L1_bnbusd
live_feeds_to_redis_pipeline/ftx_L1_btc-perp
live_feeds_to_redis_pipeline/ftx_L1_btcusd
live_feeds_to_redis_pipeline/ftx_L1_dogeusd
live_feeds_to_redis_pipeline/ftx_L1_dotusd
live_feeds_to_redis_pipeline/ftx_L1_ethusd
live_feeds_to_redis_pipeline/ftx_L1_lunausd
live_feeds_to_redis_pipeline/ftx_L1_solusd
live_feeds_to_redis_pipeline/ftx_L1_xrpusd
live_feeds_to_redis_pipeline/ftx_blotter_avaxusd
live_feeds_to_redis_pipeline/ftx_blotter_bnbusd
live_feeds_to_redis_pipeline/ftx_blotter_btc-perp
live_feeds_to_redis_pipeline/ftx_blotter_btcusd
live_feeds_to_redis_pipeline/ftx_blotter_dogeusd
live_feeds_to_redis_pipeline/ftx_blotter_dotusd
live_feeds_to_redis_pipeline/ftx_blotter_ethusd
live_feeds_to_redis_pipeline/ftx_blotter_lunausd
live_feeds_to_redis_pipeline/ftx_blotter_solusd
live_feeds_to_redis_pipeline/ftx_blotter_xrpusd
redis_to_postgres_pipeline/redis-to-postgres-deployment
Rajan Subramanian
03/09/2022, 11:51 PMRajan Subramanian
03/10/2022, 12:20 AMKevin Kho
Rajan Subramanian
03/10/2022, 12:50 AMRajan Subramanian
03/10/2022, 12:51 AMRajan Subramanian
03/10/2022, 5:55 PMKevin Kho
Rajan Subramanian
03/10/2022, 6:04 PMRajan Subramanian
03/10/2022, 6:04 PMRajan Subramanian
03/10/2022, 6:06 PMRajan Subramanian
03/10/2022, 6:07 PMKevin Kho
Rajan Subramanian
03/10/2022, 6:10 PMRajan Subramanian
03/10/2022, 6:22 PMRajan Subramanian
03/10/2022, 6:45 PMRajan Subramanian
03/10/2022, 6:45 PMKevin Kho
Kevin Kho
Rajan Subramanian
03/22/2022, 5:45 PMprefect deployment create deployment_name
if the agent hasn't started yet, its then scheduled to run once worker queue is created and run. thaksKevin Kho
Rajan Subramanian
03/22/2022, 5:52 PM