Hello, Is Prefect a good choice for non-Python based workloads? Examples: generic workload management with classic SQL Server, Oracle, Kafka and Informatica based batch and streaming data processing jobs.
Joseph Vedam
05/10/2022, 8:13 PM
i.e. just as long as the jobs can be invoked from within a Python wrapper and called from within Prefect flows?
k
Kevin Kho
05/10/2022, 8:24 PM
Surely Prefect can be used for the batch jobs. The streaming is a maybe. It depends on your use case and the latency needed. For 2.0, it would be a long run flow that continues to submit tasks. If you really need streaming, try starting with Prefect 2.0
Kevin Kho
05/10/2022, 8:24 PM
But yeah you can wrap with the flow decorator and see how that goes. Make sure to use 2.0
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.