https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Aqib Fayyaz

02/26/2022, 7:47 AM
Do we have some working example for docker sidecar on kubernetes on github?
a

Aqib Fayyaz

02/26/2022, 10:22 AM
I have registered the flow and it was register successfully with server on gke. But i am confused i have just flow and nothing else and how it build the image and pushed it to gcr as well, i mean i don't have dockerfile or something where i coped flow and custom modules etc
a

Anna Geller

02/26/2022, 10:30 AM
I didn't understand your question - can you describe your problem and what do you try to do?
a

Aqib Fayyaz

02/26/2022, 11:30 AM
ok the thing is now after trying different storage option i found this docker side car is most suitable for our case so now i want to use this. Now how can i use custom modules along with the flow in case of docker sidecar and where is dockerfile for this as well the example you shared has only flow ( a python file and nothing else with it)
and if i run that python file i mean register it using prefect register command it does register the flow as well and even builds the images and pushes to gcr even thourgh i dont have dockerfile for that.
a

Anna Geller

02/26/2022, 11:49 AM
I think you still didn't describe the actual problem. What do you try to do? Do you have some non-Python processes that you need to orchestrate? Why do you believe you need to use the sidecar containers? I honestly believe this would unnecessarily overcomplicate things for you. Can you start by describing what do you try to orchestrate with Prefect? Maybe you can share example flows you try to orchestrate?
a

Aqib Fayyaz

02/26/2022, 11:53 AM
ok the thing is i need to use shared storage (filestore instance) which is already mounted on other pods and working for them as well now i want my flow to run inside the pod which will have shared volume attached and my flow result will be stored to that shared volume and this docker sidecar gives you cutom job template with which i can mount shared voulme with the pod
simple i need to access pvc (filestore instance so that i can store the result of my flow there not flow)
a

Anna Geller

02/26/2022, 11:58 AM
you are describing some technical solutions, but I asked you if you could describe the problem. Forget Kubernetes and GCP for a moment 😄 can you please describe what do you try to do/orchestrate with Prefect? no technology, just from a business perspective. Can you share some flows do you try to orchestrate?
a

Aqib Fayyaz

02/26/2022, 12:00 PM
😅 ok yeah we are doing some processing on our data using spark it takes data from zip file process it sends it to postgres than postgres takes it gives it to feast for its online and offline store and at the end that registry file of feast needs to go to shared volume
??
flow is registered successfully and job is created on gke but when i run the flow getting this
k

Kevin Kho

02/26/2022, 2:01 PM
I don’t know if we’d be able to help you on this cuz this is something deeper on your infrastructure setup and this is not a Prefect problem. The container can’t spin up
a

Anna Geller

02/26/2022, 3:06 PM
I agree with Kevin and based on the problem you described (running a Python data application doing feature extraction for ML) I don't understand why you would need all this Docker sidecar overhead and mounting PVs. Your flow doesn't seem to do any heavy processing or having any complex dependencies. I believe you may solve the problem you described just using local agent and spinning up server with "prefect server start --expose"
a

Aqib Fayyaz

02/27/2022, 12:14 PM
no i need to run things on gke this is managment demand!
i know need to run sidecar
3 Views