https://prefect.io logo
Channels
Powered by
v

volkerjaenisch

11/22/2021, 11:17 PM
OK here we go. I noticed that the UI of Orion is way back behind the funtionality of Prefect. When can one expect a catchup?
k

Kevin Kho

11/22/2021, 11:19 PM
Hey @volkerjaenisch, keeping the discussion to Slack threads would be preferred since it helps us revisit the message as more messages come in. We’ll still see the updates.
v

volkerjaenisch

11/22/2021, 11:21 PM
Oh sorry. Using slack to first time. Was the problem that I addressed you directly?
k

Kevin Kho

11/22/2021, 11:22 PM
Not at all! I am responsible for the community so feel free to ping me. Just pinging other Prefect employees is discouraged
v

volkerjaenisch

11/22/2021, 11:24 PM
Please help me out. Where exactly shall I post my questions to. Obviously not this thread.
k

Kevin Kho

11/22/2021, 11:25 PM
This thread definitely works. You can ask everything here.
If you want to discuss them as separate issues, could we at least move code blocks and tracebacks to the respective issues so they don’t crowd the main channel?
v

volkerjaenisch

11/22/2021, 11:48 PM
So sorry. Can you please guide me more.
I think I am not allowed to move a message between threads.
k

Kevin Kho

11/22/2021, 11:50 PM
You can just edit a post, copy everything, and paste it into a thread.
v

volkerjaenisch

11/22/2021, 11:51 PM
OK I will try.
Is there a plan to include class decorators (@flow) or instance method decorators (@task). Background is that we have lots of scientific code already organized in a strict Factory/Worker pattern. It seems possible to shape our model with a simple adapter decorators to work with orion. And that is quite cool! Will this stil be possible in the future of orion? Please keep an eye on this usecase. 
from prefect import task
def to_func(cls):
"""Decorator to transform an operator class to a
function for an orion task decorator"""
instance = cls()
return instance.run
class BaseWorker:
def out(self, todo):
print(todo)
def run(self, todo):
self.out(todo)
return todo
@task(name='WorkerA')
@to_func
class WorkerA(BaseWorker):
def out(self, todo):
print('Worker A: ' + todo)
@task(name='WorkerB')
@to_func
class WorkerB(BaseWorker):
def out(self, todo):
print('Worker B: ' + todo)
class WorkerFactory:
def __call__(self, choice):
if choice == "A":
return WorkerA
elif choice == "B":
return WorkerB
@flow()
def main():
worker = WorkerFactory()('A')
<http://logger.info|logger.info>(worker('Examine Orion!'))
worker = WorkerFactory()('B')
<http://logger.info|logger.info>(worker('Examine Orion!'))
if __name__ == '__main__':
main()
Cheers, Volker
Better?
To get more feeling of the development process of Orion I tried to install the dev-Version from GH. This ends up with 
(orion-2cnkrIXd-py3.9) volker@runner:~/workspace/ORION/orion$ prefect orion start
Starting Orion API...
INFO:     Started server process [471043]
18:03:22.344 | Started server process [471043]
INFO:     Waiting for application startup.
18:03:22.344 | Waiting for application startup.
18:03:22.344 | Agent service scheduled to start in-app
18:03:22.344 | Scheduler service scheduled to start in-app
18:03:22.344 | MarkLateRuns service scheduled to start in-app
INFO:     Application startup complete.
18:03:22.348 | Application startup complete.
INFO:     Uvicorn running on <http://127.0.0.1:4200> (Press CTRL+C to quit)
18:03:22.349 | Uvicorn running on <http://127.0.0.1:4200> (Press CTRL+C to quit)
18:03:22.386 | Scheduled 0 runs.
18:03:22.387 | Finished monitoring for late runs.
18:03:27.370 | Finished monitoring for late runs.
INFO:     127.0.0.1:45934 - "OPTIONS /graphql HTTP/1.1" 200 OK
INFO:     127.0.0.1:45934 - "POST /graphql HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:45934 - "POST /graphql HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:45934 - "POST /graphql HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:45934 - "POST /graphql HTTP/1.1" 404 Not Found
18:03:32.370 | Finished monitoring for late runs.
INFO:     127.0.0.1:45934 - "POST /graphql HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:45934 - "POST /graphql HTTP/1.1" 404 Not Found
k

Kevin Kho

11/22/2021, 11:53 PM
Thank you!
v

volkerjaenisch

11/22/2021, 11:53 PM
Orion lags behind at some crucial things like logging. Even the local logging does not work with the a4 version. When are such basic operations planed to be available. Will the UI of Orion match the UI of Prefect, and if when?
Sorry, my fault. Got a bit lost in the UI.
k

Kevin Kho

11/22/2021, 11:54 PM
No worries at all!
On logging and the UI, they will take different forms from current Prefect. We don’t have any public roadmap or timeline for them. If you need something production ready, the recommendation would still be to the current Prefect (not Orion).
I will pull someone for the other questions. Responses may be slow as US holidays are nearing
I think the people who can answer are out of the office today already so do expect a response tomorrow and feel free to ping me if you don’t get one.
On the 
dev-Version
from GH, it looks like you are running Prefect Server on the same machine and it is occupying the port
v

volkerjaenisch

11/23/2021, 12:36 AM
We will need the functionality/flexibilty of Orion. Prefect is out of question since Prefect does not allow for the generation of Flows at runtime. Prefect is absolutely cool for the KI people out there running really simply workflows. We are dealing with a scientific code with 280 usecases. Not 280 codes but one code with so many usecases. It is one of the scientific kernels for the ACTRIS SCC code which evaluates and derives the LIDAR data for Europe. Disclaimer: This is not the customer I talked about formerly. Their code is much more complex. We have a deep nested graph of arrows and boxes. Every box is an operation class and every arrow is a factory to produce the worker class (algorithm) chosen by the user. So the code is statically a DAG and for sure a DAG as any code at runtime. But the second requirement to this code is that any part (Box) of it may be replaced (at runtime) by a replacement (Plugin). This requirement has lead our code design to a component approach (ZCA). We are absolutely happy with our design. Orion will not be the hero to rescue the damsel in distress. But we had planned to come up with a monitoring system like Prefect for this code. Prefect will not do the job since we have nested Flows at runtine. Orion is exactly what we need.
graph.svg
Thank you for the hint with the occupied port. This may be quiet possible. And this should be shouted by Orion into my face.
z

Zanie

11/23/2021, 12:44 AM
Sounds really cool! Logging in Orion is actually far more powerful (you can configure individual logging classes, custom formats, etc), we just haven’t decided how we want to store the logs server-side. We’re in the process of designing that right now. I’m not sure about a class decorator. Classes are stateful, and I wouldn’t want to mislead users as state will behave differently as concurrency and distributed execution is introduced. You can use a class to return static methods wrapped as tasks? If you’re just using them as composable function factories and not making use of state, you shouldn’t need a class level decorator.
v

volkerjaenisch

11/23/2021, 12:44 AM
Yep. Thank you so much. The occupied port was the issue.
z

Zanie

11/23/2021, 12:45 AM
With the release of the schematics today in the UI, we’re moving quickly towards parity there. Much of what’s missing from the UI doesn’t exist server-side yet. As we introduce things server-side (e.g. “run configs”) we’ll be adding them to the UI as well.
v

volkerjaenisch

11/23/2021, 12:55 AM
@Zanie I agree with you all the way. The first thing we smash into our customers brains is to never store state in an operational class. We have a strict seperation of data classes and operational classes. Op-Classes modify data classes. Data classes are dumb but may have properties to convert unit and so. Without this you will never be able to do easy unit testing (pickle before, pickle after) of a certain part of code. I am not strictly asking for a class decorator. You are absolutely right this may mislead. I just beg you to keep in mind people like us having a well structured component designed code heavily armed with OOP that also like to participate on your truly glorious way to Orion. I volunteer to be a tester on this part of Orion - since it is so COOL. Thank you so much for Prefect and Orion.
Another few questions I would like to ask. 1. Since in Orion there is no graph know before the process is running. a. How do you visualize the graph structure? b. How do you optimize e.g. parallelism? 2. When is a feature freeze to be expected? Aka a beta Version? 3. Will the overall Operation of Orion remain so lean (I appreciate not having Docker containers)? 4. Will you offer support for self hosted Orion installations? 5. Are there known limitations to Orion?
z

Zanie

11/23/2021, 4:26 PM
1a. There is an intention to investigate a dry run mode for visualization of the graph before runtime. Otherwise, see https://www.prefect.io/blog/introducing-radar/ 1b. Parallelism is eager right now. Since the DAG is dynamic, you can actually control the parallelism more concretely. We’ll be certain to ensure people can do things like depth-first execution. 2. https://orion-docs.prefect.io/faq/#when-will-orion-be-released 3. Yes, we do not want to ask users to use docker-compose to run the server. 4. I cannot promise, but we have offered support of this manner in the past. 5. I’m not sure yet! We’re still learning. You’d have to be more specific. I suggest reading the rest of the FAQ as it covers some of these things.
v

volkerjaenisch

11/23/2021, 8:34 PM
Thank you for the fast and thoroughly answers. Today the decision was made to give Orion a chance. Followed up by prefect as plan B.
But another question comes around. Prefect offered a functional as well as an imperative approach. Will there be support for an imperative approach for Orion as well? We would like that really much.
4 Views
Powered by