Hello, Is Prefect a good choice for non-Python based workloads? Examples: generic workload management with classic SQL Server, Oracle, Kafka and Informatica based batch and streaming data processing jobs.
Joseph Vedam
05/10/2022, 8:13 PM
i.e. just as long as the jobs can be invoked from within a Python wrapper and called from within Prefect flows?
k
Kevin Kho
05/10/2022, 8:24 PM
Surely Prefect can be used for the batch jobs. The streaming is a maybe. It depends on your use case and the latency needed. For 2.0, it would be a long run flow that continues to submit tasks. If you really need streaming, try starting with Prefect 2.0
Kevin Kho
05/10/2022, 8:24 PM
But yeah you can wrap with the flow decorator and see how that goes. Make sure to use 2.0