Last noob question before I actually run a POC I promise
"Where is an example of prefect task that just runs a pre-built Docker container."
I've read the docs extensively, and I think this is covered with LocalExecutor + DockerRun, but the examples are confusing me -- they seem to all be set up for executing some prefect flow python code from the flow run on various environments.
Here is my use case:
• I have data transform code written in Julia, in a docker container already built with an environment to execute that Julia code, along with permissions to access s3 for read/write on either end of the execution.
• I just want to schedule the running of this container (Fargate, EKS?)
• Does Prefect need to be installed on the container?
• Does the flow code need to execute some python commands in the container to start it off?
This probably seems obvious, but its not to me. I want to use Prefect as an orchestrator for many kinds of jobs (Spark, julialang, shell calls to internal team services), and the API is full of environment-specific tasks like "AWS Tasks", "DBT Tasks", "Shell Tasks", etc. I am confused if generic docker containers can be run or if only the kind of tasks from the API documentation can be run.
Thread in Slack Conversation