New to Prefect, wondering if anyone is using Prefect to manage data flows with tasks written in Node, Java or Clojure.
And how would you do it? Shell task that kicks the Node or Java/Clojure code? Or Container autorun like task?
11/19/2020, 9:18 PM
Hi @Mike Grabenstein! Welcome 😄
I’d probably recommend the ShellTask as it will handle creating a subprocess & recording logs in the Flow nicely for you
We do also have Tasks in the task library that run containers or Kubernetes jobs, if you would prefer to use those
Let me know if you have any feedback or questions!
11/19/2020, 9:24 PM
Like to see a prefect.tasks.core.function.FunctionTask* of some sort be able to call over to Java via Jython. That would be slick.
Any support for SQS, or a broker like RabbitMQ? @Dylan
11/19/2020, 9:37 PM
Would you mind opening an issue for the Jython task?
We support event-driven flows through API calls, but we don’t have a steaming solution at this time. We do have a PIN (Prefect Improvement Notice) that discusses it and we’d love to hear your thoughts