Is it possible to create your own executor, esp., one in a different language? I am concerned about how Python centric everything looks - I’m looking for the ability to run both Python and Julia tasks (the Julia tasks very well might be calling into Python via PyCall, so I’d need to make sure that that could be handled also)

A couple things here, but would love others to weigh in as well. First, you’ll likely get a better / more thought out response in the

#prefect-community

channel (this is just about the open source server). Second is that you theoretically should be able to implement your own executor (all you’d need to do is inherit from the

Executor

class at

prefect.executors.base

and implement those methods. Executors are responsible for submitting work to an engine that schedules the work accordingly. Is there an engine that exists in the Julia ecosystem (I know very little about Julia so bear with me) that you’d like to leverage? If you’re just looking to execute Julia code, you may be interested in creating a

Julia

task that accepts your python objects as arguments, interfaces with Julia to execute some code, then returns the Python objects. I’m sure the exact process may vary depending on how you need the interface between the languages to work, but I think that’d be a great starting place!

Well, inheriting from a Python class implies that you have to use Python. There is a package, Dagger.jl, which is closely patterned off of Dask, implemented in pure Julia.

Currently creating your own executor is not supported (it's doable, but not supported as the interface may and will change). Nothing about prefect's design requires Python to work (all api calls are just json), but in practice implementing full driving support for a language outside python is a large amount of work. Rather, I'd recommend using something like pyjulia to call out to Julia as needed, but doing the plumbing work in python. With pycall you might be able to do it all technically from within julia itself (as a wrapper around the python library), but this would be up to you.

OK.

pyjulia

is kind of a weird reverse

PyCall

(it actually depends on

PyCall

), and I would prefer to stick with Julia and use

PyCall

when necessary, but if that’s all that is currently possible, it will have to do.