What are good ways to call functions in other languages from

Question

What are good ways to call functions in other languages from flows I have some data actions I want to do in Julia

merlin · Answer

so far I can think of run a docker container define julia environment in there import the python julia package and call the julia function from python call julia script from a shell command There must be pro con to these approaches

Nate · Answer

This is something I ve been thinking about lately as well one related approach that s worked for me for a in a side project that I d like to formalize in the prefect docker collection is just basically what you mentioned just running an arbitrary container as a context manager in a flow and sending it commands to run i d love to implement something more general so basically like but for any docker container I don t know that there s a lot of precedence for this in the context of prefect but I m excited about the possibilities

Nate · Answer

I haven t gone deeply into this idea yet where some difficulties may be in the details but I m not sure I see an immediate drawback to the approach of spawning arbitrary containers or just running shell commands as you mention as a part of your prefect flow as long as the execution environment can perform that work

merlin · Answer

I m less apprehensive since you describe it the same way Starting to appreciate the observability part of orchestration vs managing every little step in flows

Zanie · Answer

fwiw the `DockerContainer` and `Process` infrastructure abstractions that we use for flow runs are totally valid for this purpose

Zanie · Answer

```python c from prefect infrastructure import DockerContainer DockerContainer command= julia version image= julia latest run 19 29 24 703 | INFO | prefect infrastructure docker container Pulling image julia latest 19 29 25 406 | INFO | prefect infrastructure docker container Creating Docker container with auto generated name 19 29 25 473 | INFO | prefect infrastructure docker container Docker container fervent moore has status created 19 29 25 879 | INFO | prefect infrastructure docker container Docker container fervent moore has status exited julia version 1 8 5 19 29 25 911 | INFO | prefect infrastructure docker container Docker container fervent moore has status exited ```

Zanie · Answer

Nate I d be careful not to reinvent this I have plans to split `run` into `start` and `wait` methods as well which would make a context manager trivial

Nate · Answer

that s a good point and thanks for heads up I just got excited and forgot we can use infra blocks that way smile

merlin · Answer

this is awesome good straight line simplicity i would have spent a lot of time on this still gotta choose mounting a volume vs building the container with my julia code