https://prefect.io logo
Channels
Powered by
r

Rajan Subramanian

03/02/2022, 5:36 PM
hello, i recently got some piplelines deployed on orion UI. How do i stop a task run? i see two similar tasks being run concurrently and want to stop one of them
k

Kevin Kho

03/02/2022, 5:38 PM
Hi @Rajan Subramanian, let me check with the team if the cancellation endpoint is attached for the UI already
r

Rajan Subramanian

03/02/2022, 9:53 PM
Hello, i have recently deployed 3 pipelines to prefect orion2 dashboard. it seems only two pipelines are running at once rather than 3. The third one seems to be in pending state: feed_to_redis_pipeline.py
Copy code
@task(
    name="run_streamer_in_shell",
    description="runs each of the exchange.py file as a separate shell task",
    retries=3,
    retry_delay_seconds=10,
)
def run_streamer_in_shell(command: Task):
    subprocess.run(command, shell=True)


@flow(name="live_feeds_to_redis_pipeline", task_runner=DaskTaskRunner())
def run_all_tasks(shell_tasks: List[str]):
    for path in shell_tasks:
        cmd = build_command(path)
        run_streamer_in_shell(command=cmd)
and heres my deployment.py
Copy code
STREAM_TASKS = ['path_to_ex1.py", "path_to_ex2.py, path_to_ex3.py"]
DeploymentSpec( name="feed-to-redis-deployment", _flow_location_="feed_to_redis_pipeline.py", tags=["exchange-streamer", "raj"], parameters={"shell_tasks": STREAM_TASKS}, ) ``````
The build command is just a shell command with "python path/to/python/file.py"
k

Kevin Kho

03/02/2022, 9:55 PM
It seems Dask is only seeing 2 cores? Maybe you can be more explicit with your workers in the DaskTaskRunner and see if it helps? Dask infers but it’s off sometimes
About the cancellation, we don’t have it in the UI and we dont have interrupt style cancellation yet. Still thinking about it.
r

Rajan Subramanian

03/02/2022, 9:56 PM
@Kevin Kho, thanks i will try and let u know the status of it
k

Kevin Kho

03/02/2022, 9:57 PM
Acutally maybe you should just use the ConcurrentTaskRunner . It might be better
r

Rajan Subramanian

03/02/2022, 9:58 PM
@Kevin Kho, for now i am following @Anna Geller advise and using:
Copy code
prefect orion database reset -y
👍 1
@Kevin Kho, when you say that dasktaskrunner assumes 2 cores, if i have 20 files that i am looping over and executing, how many cores should i specify?
k

Kevin Kho

03/02/2022, 10:01 PM
How many cores you have in your machine that is executing? Maybe try the ConcurrentTaskRunner first
r

Rajan Subramanian

03/02/2022, 10:08 PM
@Kevin Kho, i have 10 cores in my machine. In these taskrunners, is there any example how to specify the number of cores?
k

Kevin Kho

03/02/2022, 10:08 PM
For Dask or Concurrent Task Runner?
Copy code
# Use 4 worker processes, each with 2 threads
DaskTaskRunner(
    cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}
)
r

Rajan Subramanian

03/02/2022, 10:15 PM
@Kevin Kho thanks a lot
its working
k

Kevin Kho

03/02/2022, 10:16 PM
Nice!
r

Rajan Subramanian

03/02/2022, 10:23 PM
at the moment, an execution of a prefect deployment has this running
Screen Recording 2022-03-02 at 5.22.40 PM.mov
looks good i guess 🙂
k

Kevin Kho

03/02/2022, 10:25 PM
💰💰💰
r

Rajan Subramanian

03/02/2022, 10:25 PM
somewhat im confused about is, when i execute all these tasks in parallel, is it happening concurrently? i rather not have another task get delayed just because another work didn't finish his task. so what i would prefer is just all these tasks executing at the same time rather than concurrently although one implies the other
k

Kevin Kho

03/02/2022, 10:28 PM
They are concurrent yep. As long as you have an available worker, it will pick up the work. It will get delayed if you have more tasks than workers
r

Rajan Subramanian

03/03/2022, 9:49 PM
@Kevin Kho, hmm this is kind of problematic if its concurrent execution as i dont want another task to depend on execution of another task. Question i am streaming currency data for orderbook and currency data for blotter. so if have one currency, thats two stream files i have. so, to prevent concurrent execution, is it better to just create 2 separate pipeline? what if i have 50 currencies. thats 100 pipelines just for one exchange.
k

Kevin Kho

03/03/2022, 10:03 PM
I am confused. Isn’t concurrency for independent tasks? So if task1 and task2 don’t need each other, you can just run them at the same time? If you want sequential execution, just change to the SequentialTaskRunner instead of the default ConcurrenctTaskRunner
r

Rajan Subramanian

03/03/2022, 10:09 PM
@Kevin Kho, yes they are independent. So if i ran Task1 and Task2, i dont want task2 to be delayed at all just because task1 is delayed. i am daskTaskRunner
would u recommend to stick with my original method or create parallel runs?
k

Kevin Kho

03/03/2022, 10:12 PM
The DaskTaskRunner should already run them at the same time. Do you see task2 is delayed? Because that would be a bug that we would investigate if you can give a minimum reproducible example. It should be parallel
r

Rajan Subramanian

03/03/2022, 10:12 PM
no i dont see any delays at the moment but will let you know if i do observe @Kevin Kho
@Kevin Kho, i have now deployed 6 files. 5 are executing, but 6th one is pending. should i increase the number of workers or threads here?
k

Kevin Kho

03/04/2022, 5:56 PM
I guess so if you have enough cores
r

Rajan Subramanian

03/04/2022, 7:01 PM
ok then i will split the deployments into separate deployments that way i dont have to worry about scalability issues
@Kevin Kho, so another question, i currently have 4 deployments associated with a worker-queue. If i want to add additional deployments, i type
Copy code
prefect deployment create ftx_deployment
however this isn't being picked up by the orion ui. doesn't the worker queue listen to all activate deployments if you dont specify a worker queue to a deployment? initially, when i created deployments, i merely created a worker queue and did
Copy code
prefect agent start <'uiud in string format
and went to the ui and saw all the active deployments there. but after starting teh agent and deploying more files, the ui isn't picking it up unless i stop everything and reset the database
k

Kevin Kho

03/07/2022, 4:08 PM
Will have to check with the team on this one
r

Rajan Subramanian

03/07/2022, 4:09 PM
@Kevin Kho, k thanks
i guess my main question is it possible to add additional deployments to the same worker queue without stopping all deployments and resetting the database. thanks
k

Kevin Kho

03/07/2022, 4:11 PM
I believe you should be able to but will confirm. That would be pretty painful if you couldn’t
✅ 1
I think we’d need a reproducible example for this to really look into it because it should be working
r

Rajan Subramanian

03/09/2022, 6:24 PM
@Kevin Kho, thanks for your help. question, is it possible to add multiple deploymentspecs in a single deployment file?
k

Kevin Kho

03/09/2022, 6:41 PM
Yes you can
r

Rajan Subramanian

03/09/2022, 6:42 PM
so in the future, if i add more deployment specs, does prefect automatically pick up these deployment specs? or do i do a
Copy code
prefect create deployment.py
again where it contains the new deployment spec and previous deployment specs
k

Kevin Kho

03/09/2022, 6:44 PM
You need to re-create to add the new ones
r

Rajan Subramanian

03/09/2022, 6:45 PM
oh ok thanks
This isn't being recognized...steps: 1. prefect orion start 2. prefect deployment create binance/orderbook_deployments 3. prefect deployment create binance/blotter_deployments 4. prefect deployment create redis_to_postgres_deployment After the above steps, i then do
Copy code
prefect work-queue create feed-to-postgres-agent
prefect agent start 'uiud' obtained from previous step
I then go to the ui, go to deployments and hit quick run and see the runs as such: I then go to my terminal and type:
Copy code
prefect deployment create ftx/blotter_deployment.py
which creates 10 deployments from teh specs i have in that file. i then go to my terminal and type
Copy code
prefect deployment ls
and see the following:
Copy code
live_feeds_to_redis_pipeline/binance-blotter for btcusdt
live_feeds_to_redis_pipeline/binance_blotter_ethusdt
live_feeds_to_redis_pipeline/binance_l1_btcusdt
live_feeds_to_redis_pipeline/binance_l1_ethusdt
live_feeds_to_redis_pipeline/ftx-blotter for avaxusd
live_feeds_to_redis_pipeline/ftx-blotter for bnbusd
live_feeds_to_redis_pipeline/ftx-blotter for btc-perp
live_feeds_to_redis_pipeline/ftx-blotter for btcusd
live_feeds_to_redis_pipeline/ftx-blotter for dogeusd
live_feeds_to_redis_pipeline/ftx-blotter for dotusd
live_feeds_to_redis_pipeline/ftx-blotter for ethusd
live_feeds_to_redis_pipeline/ftx-blotter for lunausd
live_feeds_to_redis_pipeline/ftx-blotter for solusd
live_feeds_to_redis_pipeline/ftx-blotter for xrpusd
redis_to_postgres_pipeline/redis-to-postgres-deployment
I refreshed my ui many times and dont see that being recognized. what can i do?
k

Kevin Kho

03/09/2022, 9:08 PM
So you are expecting 10 deployments but only seeing 5 in the UI?
Do the other ones have schedules on them?
Deployments without schedules are not shown in the UI by default
r

Rajan Subramanian

03/09/2022, 10:06 PM
oh
no there are no schedules for any of them
how would i run deployments with no schedules associated with them?
k

Kevin Kho

03/09/2022, 10:15 PM
On the UI at the bottom you should be able to “show all” and it will show the ones without schedules and than you can Quick Run
r

Rajan Subramanian

03/09/2022, 10:17 PM
ok let me see
where is it? i dont see it
@Kevin Kho
@Kevin Kho, your t houghts on this approach, i created one pipeline, and another deployment file with 10 deployment specs, that calls the pipeline file with parameter for each deployment.
curious if thats the right way to call a pipeline with multiple parameters
k

Kevin Kho

03/09/2022, 10:22 PM
Will ask the UI team. Yeah that sounds good to me if you have multiple stuff to fetch that are independent. But if it gets too big, you can just loop over a list in the flow and take a list of things to process instead
r

Rajan Subramanian

03/09/2022, 10:26 PM
@Kevin Kho, it seems on the filter section, its defaulting to two filters: show flow runs by start date, show flow runs by start date when i reset the filters and go back to the. ui, it defaults to those two basic filters again
k

Kevin Kho

03/09/2022, 10:28 PM
You mean the 
frn:1d
and 
fru:1d
at the top? Are you still trying to find the flows with scheduled or is this something else? Still waiting for the UI team on that
Chatted with them and the button did disappear as part of a refactor. It is being replaced with something else, for now if you remove the time filter up top, you can see it:
I have two flows here without schedules
r

Rajan Subramanian

03/09/2022, 10:40 PM
hey i managed to fix it
i chose the filters for the two flow runs and can see everything thanks
👍 1
yea its weird, i see deployments 25 when i do prefect deployment ls, but only 20 are being deployed and i cant view the other 5 deployments
there must be a way to view all deployments in a single view and all deployments not run that way its easier to navigate and see which deployments have been run adn which haven't.
this is what im working with right now, the pane doesn't show 5 deployments
Copy code
live_feeds_to_redis_pipeline/binance-blotter_btcusdt
live_feeds_to_redis_pipeline/binance_blotter_ethusdt
live_feeds_to_redis_pipeline/binance_l1_btcusdt
live_feeds_to_redis_pipeline/binance_l1_ethusdt
live_feeds_to_redis_pipeline/ftx_L1_avaxusd
live_feeds_to_redis_pipeline/ftx_L1_bnbusd
live_feeds_to_redis_pipeline/ftx_L1_btc-perp
live_feeds_to_redis_pipeline/ftx_L1_btcusd
live_feeds_to_redis_pipeline/ftx_L1_dogeusd
live_feeds_to_redis_pipeline/ftx_L1_dotusd
live_feeds_to_redis_pipeline/ftx_L1_ethusd
live_feeds_to_redis_pipeline/ftx_L1_lunausd
live_feeds_to_redis_pipeline/ftx_L1_solusd
live_feeds_to_redis_pipeline/ftx_L1_xrpusd
live_feeds_to_redis_pipeline/ftx_blotter_avaxusd
live_feeds_to_redis_pipeline/ftx_blotter_bnbusd
live_feeds_to_redis_pipeline/ftx_blotter_btc-perp
live_feeds_to_redis_pipeline/ftx_blotter_btcusd
live_feeds_to_redis_pipeline/ftx_blotter_dogeusd
live_feeds_to_redis_pipeline/ftx_blotter_dotusd
live_feeds_to_redis_pipeline/ftx_blotter_ethusd
live_feeds_to_redis_pipeline/ftx_blotter_lunausd
live_feeds_to_redis_pipeline/ftx_blotter_solusd
live_feeds_to_redis_pipeline/ftx_blotter_xrpusd
redis_to_postgres_pipeline/redis-to-postgres-deployment
hence 24 deployments. UI says 24 deployments but only 20 are showing
i actually tried resetting the database and tried again, the task runs disappears from the task pane.
k

Kevin Kho

03/10/2022, 12:32 AM
Need to ask the UI team how to view more or if you have to filter. I think losing task runs is expected if you restart the database? Since you lose data when you do that?
✅ 1
r

Rajan Subramanian

03/10/2022, 12:50 AM
@Kevin Kho disregard my previous statement, i had an error in my code
@Kevin Kho, no i meant the deployments are disappearing from the task pane. I can still the tasks.
@Kevin Kho, im not able to view my deployments. I just deployed 30 deployments, but can only view 20 on the UI. all are associated with a single pipeline "live_feeds_to_redis_pipeline" which i filtered for above on the UI and it accurately shows 20 deployments. but when i deploy 10 more, those 10 dont get populated. although it says there are 30 deployments. is there anyway to make the UI easier to use? i.e show all deployments when u filter by a flow name
k

Kevin Kho

03/10/2022, 6:01 PM
Will raise the feedback but I think you can name the deployment differently to filter on that?
r

Rajan Subramanian

03/10/2022, 6:04 PM
well i have it named as ftx_blotter_currency1, ftx_blotter_currency2
so ideally if i filter on ftx_blotter, i want it to show other deployments associated with that regex pattern
right now i have named all my deployments with a single name and i try to differentiate it from the tags but ideally the deployment name should be unique to each deployment and the filter option should have an option to do an approximate search
another thing is, i want the ability to stop a deployment either from the UI or from the cli. which ever method, but a command that either stops a task run or removes a deployment from the list of deployments i have there
k

Kevin Kho

03/10/2022, 6:09 PM
Definitely understand. Will raise all the feedback.
r

Rajan Subramanian

03/10/2022, 6:10 PM
yea ideally something like prfect deployment remove deployment_name, or prefect deployment remove deployment_name work_queue_name to remove deployment from a workqueue
ok nm i cant create same names for each deployment. has to be unique
yea definitely teh filers need to be improved. not all deployments show up in a single page. i had to manually filter out each deployment id and hit run.
is there an option to run all deployments?
k

Kevin Kho

03/10/2022, 6:58 PM
I asked them to look into pagination for the flows not showing. Run all deployments though is not a concept in Prefect 1 or Prefect 2. I think the easiest way to achieve it is to have a flow or script that initiates these runs by hitting the REST API
The team said this is all helpful feedback and some of it should be helped by changes rolled out next week
r

Rajan Subramanian

03/22/2022, 5:45 PM
hey, any possibility of allowing a flag -r or --run flag when one does
Copy code
prefect deployment create deployment_name
if the agent hasn't started yet, its then scheduled to run once worker queue is created and run. thaks
k

Kevin Kho

03/22/2022, 5:51 PM
Could you make a new post for this one in community so i can point to it and raise it for the engineers to see?
r

Rajan Subramanian

03/22/2022, 5:52 PM
sure
12 Views
Powered by