Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

What is the  development workflow for code on remote infrastructure (like Prefect-managed worker pools) ?

<https://docs.prefect.io/v3/deploy/infrastructure-examples/managed|Run flows on Prefect Managed infrastructure - Prefect> is the closest development workflow I could find. However, my problem is when I save code locally, the remote worker has no idea of the changes. I have to push to remote storage (like `git push` or `aws s3 cp &lt;src&gt; &lt;dst&gt;` for example), then do `prefect deploy`
Is there any better workflow ?

hi <@U06KP9C16N7> are you using .deploy or prefect.yaml

the answer is you tell the deployment where to fetch the code at runtime via either
• `pull` step (for prefect.yaml)
• `from_source` (for .deploy)
if you're baking code into your image, its a slightly different story, but that doesnt sound like your case here

so yeah you have to `git push` or otherwise push your code to the remote storage, bc the deployment only knows how to pull from the remote location you specified

but <https://docs.prefect.io/v3/deploy/infrastructure-concepts/prefect-yaml#the-push-action|we have a step> for pushing to buckets like s3 already if you're using prefect.yaml

Hey Nate, thanks for chiming in!
We do use `prefect deploy` (this talks to a remote prefect instance) and have this in our prefect.yaml (as you mentioned):
```pull:
- prefect.deployments.steps.git_clone:
    repository: <https://github.com/org/repo>
    credentials: '{{ prefect.blocks.github-credentials.gh-creds }}'```
Thus, for every tiny code change that we would like to test, we save the flie (Ctrl+S), `git push`, then `prefect deploy` to instantiate a flow run and see the output. It's not convenient so we thought (dev team and I) if there's something better. I will look into `push` step.

> if you're baking code into your image, its a slightly different story, but that doesnt sound like your case here
Right, it's not our case here. We have a remote k8s worker that we use during development with a custom image.

yeah so it’s up to you what’s convenient, but generally you can use the prebuilt push_to_s3 step or just run_shell_script to run arbitrary bash like aws s3 whatever in your push step so that runs upon prefect deploy

I see, thanks! btw, where can I find the documentation to all steps ? I couldn't find `push_to_s3`

i linked docs to that in the message before last

Oh ok. Yeah I checked it but I thought it didn't enumerate all options/steps.
Thanks again

yeah there might not be 100% coverage on all those steps, but a useful detail if you want to look under the hood is that those steps are just fully qualified function names, so {package}.deployments.steps.{function name}