Hello 🙋 Today i introduced our data-scientists to Prefect, and their main question was - what would be the easiest way for them to run their custom code (with all its various dependencies) from the flows they create Is containerizing their code in a docker image and pushing it to a remote repository (like ECR) the only reasonable way to achieve that? or could we conceivably just fetch the code from Github, and install the necessary dependencies as actual tasks in the flow, before executing it? (obviously it adds complexity and run-time to the flow, but are there any other reasons this might not work or be a bad idea?) (the main reason dockerizing and pushing to ECR is inconvenient for them is because they can't do it themselves, and we don't have a lot of DevOps resources that can easily provide them with a complete CI/CD pipeline for every new project they wanna test out)

Couldn't they run it locally via prefect run -n, and the Docker container and flow is build against the local Docker engine (https://docs.prefect.io/orchestration/recipes/configuring_storage.html). We use ECR but when it's successfully reviewed and merged via the CI/CD. All local testing uses Docker against dev and staging warehouses, which are flipped with environs that the data scientists and engineers have in their ~/.profile.

@Jason if they run it locally then how exactly would it run on schedule?

Hi Tom, I am trying to solve the same problem. I have a repo for prefect itself and then repos for the DS projects. Each have virtual environments setup on the machine. I have simple flows setup that either 1. Add the other repo to the path and then import functions from the DS repo and run (you can set the python environment to use in the deployment) 2. Use subprocess run to run a command line call for scripts (specifying the correct python environment in the call) Our repos push to that machine whenever there is a commit into master

Hey other Tom 😁 can you clarify what you mean by "the machine"? What machine?

sorry, just a remote server 🙂

On which you have a local prefect agent running?

Yeah exactly, I think you can do the same thing with K8s though?

@Tom Thurstan in k8s there is no “the machine”. there are ad-hoc

k8s jobs

that prefect creates per flow (which then disappear into non-existence once finished). So they must either already have the necessary code & dep’s on their designated Docker container (only possible if using a custom Docker image), or they must get them after the job starts --- and this is what i’m trying to ask about.

Ah I see, sorry my misunderstanding, if you can’t tell I’m working on this from the DS side 😂

no worries, these types of complications & confusions is exactly what i’m trying to save my DS’s from having to deal with, so they can focus on actual DS work rather than on building Dockers and deploying them to ECR