Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

What are good ways to call functions in other languages from flows? I have some data actions I want to do in Julia.

so far I can think of:

• run a docker container, define julia environment in there
• import the python julia package, and call the julia function from python
• call julia script from a shell command
There must be pro/con to these approaches?

This is something I've been thinking about lately as well, one related approach that's worked for me (for a <https://github.com/zzstoatzz/pychess/blob/main/engine.py#L41|toy example> in a side project that I'd like to formalize in the prefect-docker collection) is just basically what you mentioned, just running an arbitrary container as a context manager in a flow and sending it commands to run

i'd love to implement something more general, so basically like <https://github.com/kubernetes-client/python/blob/b313b5e74f7cf222ccc39d8c5bf8a07502bd6db3/examples/pod_exec.py#L82|this> but for any docker container

I don't know that there's a lot of precedence for this in the context of prefect, but I'm excited about the possibilities

I haven't gone deeply into this idea yet, where some difficulties may be in the details, but I'm not sure I see an immediate drawback to the approach of spawning arbitrary containers (or just running shell commands as you mention) as a part of your prefect flow, as long as the execution environment can perform that work

I'm less apprehensive since you describe it the same way. Starting to appreciate the observability part of orchestration vs. managing every little step in flows.

fwiw the `DockerContainer` and `Process` infrastructure abstractions that we use for flow runs are totally valid for this purpose

```python -c 'from prefect.infrastructure import DockerContainer; DockerContainer(command=["julia", "--version"], image="julia:latest").run()'
19:29:24.703 | INFO    | prefect.infrastructure.docker-container - Pulling image 'julia:latest'...
19:29:25.406 | INFO    | prefect.infrastructure.docker-container - Creating Docker container with auto-generated name...
19:29:25.473 | INFO    | prefect.infrastructure.docker-container - Docker container 'fervent_moore' has status 'created'
19:29:25.879 | INFO    | prefect.infrastructure.docker-container - Docker container 'fervent_moore' has status 'exited'
julia version 1.8.5
19:29:25.911 | INFO    | prefect.infrastructure.docker-container - Docker container 'fervent_moore' has status 'exited'```

Nate I’d be careful not to reinvent this. I have plans to split `run` into `start` and `wait` methods as well which would make a context manager trivial.

that's a good point, and thanks for heads up. I just got excited and forgot we can use infra blocks that way :smile:

this is awesome, good straight line simplicity. i would have spent a lot of time on this. still gotta choose mounting a volume vs. building the container with my julia code.