Can Prefect Orion Server APIs be used to build and...
# best-practicesb
Binoy Shah
05/19/2022, 1:22 PMCan Prefect Orion Server APIs be used to build and orchestrate workflows with Java Developer knowledge. We dont have python developers in other teams, can Java developers build their own dockerized jobs and have them scheduled/orchestrated via Prefect Orion ?
a
Anna Geller
05/19/2022, 1:42 PM
It depends :)
if those Java developers would be willing to e.g. write a simple Python flow like this one triggering a Shell command running e.g. a Java pipeline, then the answer is yes
Copy code
from prefect import flow
from prefect_shell import shell_run_command
@flow
def example_shell_run_command_flow():
return shell_run_command(command="ls .", return_all=True)
if __name__ == "__main__":
example_shell_run_command_flow()Anna Geller
05/19/2022, 1:42 PM
the command is pretty much the only thing they would need to adjust here
b
Binoy Shah
05/19/2022, 1:43 PMIs it possible to create a full flow via APIs ? if yes, then they can execute OpenAPI based api calls to build and configure a Job run, if we use kubernetes runner, can it be done ?
a
Anna Geller
05/19/2022, 1:45 PM
it certainly is, but running a flow as above would be easier
Anna Geller
05/19/2022, 1:46 PM
you could even build some utility module allowing them to do just:
Copy code
from your_module import java_cmd_flow
if __name__ == "__main__":
java_cmd_flow(command="echo hello from Prefect")b
Binoy Shah
05/19/2022, 1:47 PMthe thing is, we have various java libraries built out of OpenAPI spec, and we can do same for Orion OpenAPI spec. These libraries will integrate from our Java Apps, and will be able to schedule jobs in more cross-language style
Binoy Shah
05/19/2022, 1:47 PMJust wanted to check if the Orion Prefect APIs are complete enough to build and orchestrate jobs via Rest APIs only
a
Anna Geller
05/19/2022, 1:52 PM
Prefect 2.0 is built in an API-first way so that in the future you could even contribute a Java orchestration API if you wanted to, but we are not that far yet to think about multi-language support, but with the current architecture it would in theory be possible
I think it may be easier to think about the problems you need to solve first (without thinking yet about the tech/language) and see how you can solve that and see how Prefect fits into that
Anna Geller
05/19/2022, 1:56 PM
e.g. which pipelines/applications do you intend to run? do you run batch/streaming data ingestion/transformation? what would be your end goal in using Prefect for that - just to schedule and trigger those or do you need retries/caching and similar functionality? if so, you would need tasks and this would be harder to do without writing Python
b
Binoy Shah
05/19/2022, 1:57 PMOur functional split is Apps teams that does some sort of rules evaluation, and Data Engg team does ELT.. both need some kind of workflow.. and we wanted to see if we can unify the solution to build DAGs/Workflows
a
Anna Geller
05/19/2022, 1:59 PM
which team are you from?
Anna Geller
05/19/2022, 2:01 PM
I'm confident you can use Prefect to consolidate that work between those two teams but I would need more info about your use case to give a better recommendation. I'm out for today but if you could describe it more (just all the info you can share about your use case here), I can think about what would be a good way to approach it with Prefect and give an answer tomorrow
b
Binoy Shah
05/19/2022, 2:20 PMI lead the Data Engg team, but I am consultant for the Java Apps teams also..
We’re at juncture, where we need to execute workflows
Binoy Shah
05/19/2022, 2:21 PMPrefect, Dagster and some other contenders which are from Non-Data engg world are on table and we need to converge on whether we can use common tool or go separate solutions
a
Anna Geller
05/20/2022, 5:11 AM
Not sure if my answer will be sufficient for you but given the information set you provided, in order for those two teams to converge on one platform, there must be some agreement and tradeoffs you need to make - either the Java team agrees to work with the Python orchestration framework (with very minimal Python code I showed you before they would need to learn to write), or your data engineering team would need to compromise and opt for some (likely GUI-based) scheduling/orchestration solution. Probably a GUI-based tool rather than Prefect/Dagster is what you are looking for but it's possible that this way nobody will end up happy and both teams feel like they made a compromise without gaining any tangible benefits. I would love to recommend Prefect to you but you need to consider the tradeoffs and choose based on which tool you eventually like most and which you will feel comfortable using.
Coming back to the original question: afaik, doing everything purely based on API calls with zero Python code might be possible in the future, but not atm.
b
Binoy Shah
05/24/2022, 1:55 PMThank you Anna
👍 1
15 Views