Would anyone be willing to show me an ETL example with basic auth? Does the script automatically live on a server some where so that it can run automatically?

Hi Thomas - what sort of auth are you referring to here?

basic auth pass and username. I have some scripts with Oauth and would need an environment to store the json files. Would you be able to do Oauth as well?

Hi @Thomas Adams, if I understand your question correctly, you'd like to store some credentials like username/password in environment variables; that's certainly doable with Prefect! https://docs.prefect.io/core/concepts/configuration.html#user-configuration should help you get started with environment variables. For an ETL example, https://docs.prefect.io/core/tutorials/etl.html#e-t-and-l should be helpful to get an understanding for how Prefect can be used to break down those tasks; adding a task to handle authorization as an upstream dependency (that returns a token or code) could be accomplished by adding a task which makes an http call to your auth provider. For that, I'd recommend the

requests

package or one of the various OAuth python libraries (https://oauth.net/code/python/ has a nice list!). Is that helpful?

Okay maybe I need to spend more time with Prefect and overall proper coding. In my scripts I am a bad coder and rarely use functions to handle a task... I know I know.. Very helpful! I'll look at it more this weekend! Any youtube vidoes out there lol?

I should also add that Prefect has a type of Task to handle sensitive data in particular, the Secret Task. You can read more about that here: https://docs.prefect.io/api/unreleased/tasks/secrets.html#secret As for videos, which sort are you looking for? Unfortunately we don't have any Prefect-specific videos.

I went straight to numpy and pandas and did not learn how to properly use functions. Time to learn functions since that's what it looks like Prefect uses a lot.

Prefect also has an Imperative API which you may find more comfortable than the functional API! You can read a bit more about that here: https://docs.prefect.io/core/concepts/flows.html#apis That may better match what you're used to coming from less functional packages!