https://prefect.io logo
Title
s

Scott Jones

02/02/2021, 6:46 PM
Is it possible to create your own executor, esp., one in a different language? I am concerned about how Python centric everything looks - I’m looking for the ability to run both Python and Julia tasks (the Julia tasks very well might be calling into Python via PyCall, so I’d need to make sure that that could be handled also)
a

Alex Cano

02/02/2021, 7:06 PM
A couple things here, but would love others to weigh in as well. First, you’ll likely get a better / more thought out response in the
#prefect-community
channel (this is just about the open source server). Second is that you theoretically should be able to implement your own executor (all you’d need to do is inherit from the
Executor
class at
prefect.executors.base
and implement those methods. Executors are responsible for submitting work to an engine that schedules the work accordingly. Is there an engine that exists in the Julia ecosystem (I know very little about Julia so bear with me) that you’d like to leverage? If you’re just looking to execute Julia code, you may be interested in creating a
Julia
task that accepts your python objects as arguments, interfaces with Julia to execute some code, then returns the Python objects. I’m sure the exact process may vary depending on how you need the interface between the languages to work, but I think that’d be a great starting place!
👍 1
s

Scott Jones

02/02/2021, 7:09 PM
Well, inheriting from a Python class implies that you have to use Python. There is a package, Dagger.jl, which is closely patterned off of Dask, implemented in pure Julia.
j

Jim Crist-Harif

02/02/2021, 7:18 PM
Currently creating your own executor is not supported (it's doable, but not supported as the interface may and will change). Nothing about prefect's design requires Python to work (all api calls are just json), but in practice implementing full driving support for a language outside python is a large amount of work. Rather, I'd recommend using something like pyjulia to call out to Julia as needed, but doing the plumbing work in python. With pycall you might be able to do it all technically from within julia itself (as a wrapper around the python library), but this would be up to you.
s

Scott Jones

02/02/2021, 7:20 PM
OK.
pyjulia
is kind of a weird reverse
PyCall
(it actually depends on
PyCall
), and I would prefer to stick with Julia and use
PyCall
when necessary, but if that’s all that is currently possible, it will have to do.
(I’ve used
PyCall
fairly frequently - it’s my preferred way of interaction with Python, to be honest!)