Joseph Vedam

05/10/2022, 8:05 PM
Hello, Is Prefect a good choice for non-Python based workloads? Examples: generic workload management with classic SQL Server, Oracle, Kafka and Informatica based batch and streaming data processing jobs.
i.e. just as long as the jobs can be invoked from within a Python wrapper and called from within Prefect flows?

Kevin Kho

05/10/2022, 8:24 PM
Surely Prefect can be used for the batch jobs. The streaming is a maybe. It depends on your use case and the latency needed. For 2.0, it would be a long run flow that continues to submit tasks. If you really need streaming, try starting with Prefect 2.0
But yeah you can wrap with the flow decorator and see how that goes. Make sure to use 2.0