Last noob question before I actually run a POC I promise
"Where is an example of prefect task that just runs a pre-built Docker container."
I've read the docs extensively, and I think this is covered with LocalExecutor + DockerRun, but the examples are confusing me -- they seem to all be set up for executing some prefect flow python code from the flow run on various environments.
Here is my use case:
• I have data transform code written in Julia, in a docker container already built with an environment to execute that Julia code, along with permissions to access s3 for read/write on either end of the execution.
• I just want to schedule the running of this container (Fargate, EKS?)
• Does Prefect need to be installed on the container?
• Does the flow code need to execute some python commands in the container to start it off?
This probably seems obvious, but its not to me. I want to use Prefect as an orchestrator for many kinds of jobs (Spark, julialang, shell calls to internal team services), and the API is full of environment-specific tasks like "AWS Tasks", "DBT Tasks", "Shell Tasks", etc. I am confused if generic docker containers can be run or if only the kind of tasks from the API documentation can be run.
Thread in Slack Conversation
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.