Thread
#prefect-server
    Scott Jones

    Scott Jones

    1 year ago
    Is it possible to create your own executor, esp., one in a different language? I am concerned about how Python centric everything looks - I’m looking for the ability to run both Python and Julia tasks (the Julia tasks very well might be calling into Python via PyCall, so I’d need to make sure that that could be handled also)
    a

    Alex Cano

    1 year ago
    A couple things here, but would love others to weigh in as well. First, you’ll likely get a better / more thought out response in the
    #prefect-community
    channel (this is just about the open source server). Second is that you theoretically should be able to implement your own executor (all you’d need to do is inherit from the
    Executor
    class at
    prefect.executors.base
    and implement those methods. Executors are responsible for submitting work to an engine that schedules the work accordingly. Is there an engine that exists in the Julia ecosystem (I know very little about Julia so bear with me) that you’d like to leverage? If you’re just looking to execute Julia code, you may be interested in creating a
    Julia
    task that accepts your python objects as arguments, interfaces with Julia to execute some code, then returns the Python objects. I’m sure the exact process may vary depending on how you need the interface between the languages to work, but I think that’d be a great starting place!
    Scott Jones

    Scott Jones

    1 year ago
    Well, inheriting from a Python class implies that you have to use Python. There is a package, Dagger.jl, which is closely patterned off of Dask, implemented in pure Julia.
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Currently creating your own executor is not supported (it's doable, but not supported as the interface may and will change). Nothing about prefect's design requires Python to work (all api calls are just json), but in practice implementing full driving support for a language outside python is a large amount of work. Rather, I'd recommend using something like pyjulia to call out to Julia as needed, but doing the plumbing work in python. With pycall you might be able to do it all technically from within julia itself (as a wrapper around the python library), but this would be up to you.
    Scott Jones

    Scott Jones

    1 year ago
    OK.
    pyjulia
    is kind of a weird reverse
    PyCall
    (it actually depends on
    PyCall
    ), and I would prefer to stick with Julia and use
    PyCall
    when necessary, but if that’s all that is currently possible, it will have to do.
    (I’ve used
    PyCall
    fairly frequently - it’s my preferred way of interaction with Python, to be honest!)