https://prefect.io
Join Slack
Hi everyone, we are using Prefect Cloud and Azure ACI's for the workers. For the ACI command for ins...
# ask-community
k
Hi everyone, we are using Prefect Cloud and Azure ACI's for the workers. For the ACI command for install I wanted to confirm I just need 
/bin/bash -c 'pip install prefect-azure && prefect worker start --pool <work-pool-name> --type azure-container-instance'
and for flows dependencies are setup on a per flow basis? i.e. I think in prefect 2 I installed all possible extra packages on the agent and flows used those. I have been having success configuring the job variables in the cloud UI and PIP EXTRA PACKAGES but I wanted to confirm that's the most efficient way. Any help much appreciated!
b
Hey Keith! Yes, to start the worker process that command is all you need. If you don't have many dependencies, and don't expect them to change often, you could keep setting them through the Cloud UI and 
job_variables.
If your dependencies change down the line, however, you would need to remember to go and manually change the packages you defined. For managing flow dependencies, typically we recommend creating a docker image that has all of them baked in (especially if you're using a containerized work pool type).
If you're using 
.deploy()
it will build a docker image that contains your flow code and dependencies (the ones listed out in your 
requirements.txt
file) by default.
k
our team is using a git repo for flow code storage not on custom docker images yet, can we specify the pip extra packages in our deployment code?
b
Yup, something like this: if name == "__main__": get_repo_info.deploy( name="my-deployment-never-pull", work_pool_name="my-docker-pool", job_variables={"env": {"EXTRA_PIP_PACKAGES": "boto3"} }, ... )
Or if you're deploying through the cli with 
prefect deploy
, you can set a 
pull
step in your 
prefect.yaml
that installs dependencies after cloning the repo
Copy code
pull:
- prefect.deployments.steps.git_clone:
    id: clone-step # needed to be referenced in subsequent steps
    repository: <https://github.com/org/repo.git>
- prefect.deployments.steps.pip_install_requirements:
    directory: "{{ clone-step.directory }}" # `clone-step` is a user-provided `id` field
    requirements_file: requirements.txt
k
Thanks Bianca, it's starting to make sense. I think I was confusing prefect 3 and 2 agents/workers
with ACI being serverless
37 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by