https://prefect.io logo
Title
m

merlin

01/19/2023, 12:43 AM
What are good ways to call functions in other languages from flows? I have some data actions I want to do in Julia.
so far I can think of: • run a docker container, define julia environment in there • import the python julia package, and call the julia function from python • call julia script from a shell command There must be pro/con to these approaches?
n

Nate

01/19/2023, 1:10 AM
This is something I've been thinking about lately as well, one related approach that's worked for me (for a toy example in a side project that I'd like to formalize in the prefect-docker collection) is just basically what you mentioned, just running an arbitrary container as a context manager in a flow and sending it commands to run i'd love to implement something more general, so basically like this but for any docker container I don't know that there's a lot of precedence for this in the context of prefect, but I'm excited about the possibilities
šŸŽÆ 2
I haven't gone deeply into this idea yet, where some difficulties may be in the details, but I'm not sure I see an immediate drawback to the approach of spawning arbitrary containers (or just running shell commands as you mention) as a part of your prefect flow, as long as the execution environment can perform that work
m

merlin

01/19/2023, 1:25 AM
I'm less apprehensive since you describe it the same way. Starting to appreciate the observability part of orchestration vs. managing every little step in flows.
:upvote: 3
z

Zanie

01/19/2023, 1:28 AM
fwiw the
DockerContainer
and
Process
infrastructure abstractions that we use for flow runs are totally valid for this purpose
python -c 'from prefect.infrastructure import DockerContainer; DockerContainer(command=["julia", "--version"], image="julia:latest").run()'
19:29:24.703 | INFO    | prefect.infrastructure.docker-container - Pulling image 'julia:latest'...
19:29:25.406 | INFO    | prefect.infrastructure.docker-container - Creating Docker container with auto-generated name...
19:29:25.473 | INFO    | prefect.infrastructure.docker-container - Docker container 'fervent_moore' has status 'created'
19:29:25.879 | INFO    | prefect.infrastructure.docker-container - Docker container 'fervent_moore' has status 'exited'
julia version 1.8.5
19:29:25.911 | INFO    | prefect.infrastructure.docker-container - Docker container 'fervent_moore' has status 'exited'
āœ… 1
šŸš€ 3
Nate I’d be careful not to reinvent this. I have plans to split
run
into
start
and
wait
methods as well which would make a context manager trivial.
n

Nate

01/19/2023, 1:32 AM
that's a good point, and thanks for heads up. I just got excited and forgot we can use infra blocks that way šŸ˜„
m

merlin

01/19/2023, 11:16 PM
this is awesome, good straight line simplicity. i would have spent a lot of time on this. still gotta choose mounting a volume vs. building the container with my julia code.
šŸ‘ 1