Hey all, just wondering if there are any good tips...
# prefect-community
Hey all, just wondering if there are any good tips / links for Prefect project structure / how to save your tasks + flows for easy source control, etc
We don't have a document on this (yet) unfortunately. Prefect is very flexible on how/where you store your flows, which is good for some users, but for new users all the possibilities can be a bit confusing. We're hoping to write up some best practices at some point to make this clearer. When possible, I generally recommend storing your flows in source control somewhere (github, bitbucket, gitlab, ...) and using the corresponding storage class associated with the source control to register your flows. For example, if using github you'd set:
Copy code
flow.storage = GitHub("your-repo", "path/to/yourflow.py")
At runtime, your flows will be pulled from github and executed, no need for an external storage mechanism. This also lets you make small changes to the flow source without requiring you to re-register the flow. This is just one opinion though, there are lots of ways people manage and organize their flows. Perhaps others in the community can give you a longer response. Is there something specific you're wondering about? A more specific question may be easier to answer.
Thanks @Jim Crist-Harif, just trying to figure out how we'd migrate some of our current jobs which are mixed between Cron, database scheduler, etc. Trying to just come up with a project structure to handle migrating my current workload. Workload consists of bash scripts calling Talend jobs, bash scripts invoking backups, calling SQL functions, etc. Initial thought is just try and consolidate everything into Prefect as-is, then start breaking apart my existing stuff where applicable into smaller Prefect tasks
Ah, that makes sense. For taking advantage of prefect features quickly, you might find the
useful for wrapping your existing shell scripts as prefect tasks (https://docs.prefect.io/api/latest/tasks/shell.html#shelltask)