https://prefect.io logo
#data-tricks-and-tips
Title
# data-tricks-and-tips
a

Angel Acosta

08/25/2022, 2:35 PM
Hello, I am using prefect cloud 2.0, I just started receiving a notification that my work queue uses a deprecated tag-based approach. What is the recommended way to create a work queue and assign a deployment to it? I am also testing using a docker container to run my flows. I am curious if there is a tutorial on this. I was able to start a prefect agent from the docker container, but It never picks up any work. (I am new to docker as well)
1
a

Anna Geller

08/25/2022, 2:40 PM
To start an agent: prefect agent start -q xxx To assign the same queue name to deployment: prefect deployment build... -q xxx
Adding --help to any CLI command provides more info on how to use those flags
a

Angel Acosta

08/25/2022, 2:57 PM
I think maybe I need to update prefect, my cli is showing "No such option: -q", I had been using prefect agent start -t dev
@Anna Geller thank you for the help, after updating I was able get that working. I also made a little progress running the agent through a docker container.. at least I was able to login to my cloud profile from the container , and then start the agent I was wanting. It did pick up work, but it failed with "RuntimeError: Could not connect to Docker."
a

Anna Geller

08/25/2022, 4:33 PM
you shouldn't run docker in docker - your agent should be a normal process - you can run it from conda environment if you need more separation
a

Angel Acosta

08/29/2022, 1:50 PM
@Anna Geller I am hoping I can get guidance on how the infrastructure should be set up. I deployed a flow to prefect cloud, I want to be able to run that flow on a schedule, but I don't want to have the agent running on my laptop 24/7. my solution was going to be setting up a server and run a docker container with the python script and agent started. Is there a more strait forward solution that I am missing?
a

Anna Geller

08/29/2022, 1:54 PM
check out this repo https://github.com/anna-geller/dataflow-ops - you can spin up an agent on ECS with a single click from GitHub Actions https://github.com/anna-geller/dataflow-ops/actions/workflows/ecs_prefect_agent.yml
a

Angel Acosta

08/29/2022, 2:02 PM
Thank you! I will look into this as a solution, the last thing I forgot to mention was that I need access to a network drive , which was the reason for running things though a server connected to our network. I would love to have everything running on AWS!
a

Anna Geller

08/29/2022, 11:38 PM
for Linux, you may check the tutorial on how to run a Prefect agent as a systemd service on Linux.
a

Angel Acosta

08/30/2022, 2:48 PM
Thank you! I made a lot of progress on this yesterday. I didn't have a clear understanding of how prefect worked under the hood, hence me trying to create a docker in docker instance. I thought I was just creating a docker container to run an agent without realizing a docker container gets created already. still wrapping my head around the infrastructure but I was able to access the prefecthq image and create a container from it, then log into my cloud account and start my agent, it picked up the flow just fine, now just figuring out storage and how to define that correctly. not sure if I just mount a drive to the container on creation or if it goes further back in the process to the flow deployment itself. I will keep you updated with progress, I really appreciate your help and guidance!
🙏 1
🙌 1
5 Views