https://prefect.io logo
Title
t

Thomas Adams

11/22/2019, 1:25 AM
Would anyone be willing to show me an ETL example with basic auth? Does the script automatically live on a server some where so that it can run automatically?
c

Chris White

11/22/2019, 1:30 AM
Hi Thomas - what sort of auth are you referring to here?
t

Thomas Adams

11/22/2019, 1:50 AM
basic auth pass and username. I have some scripts with Oauth and would need an environment to store the json files. Would you be able to do Oauth as well?
How about environment variables?
n

nicholas

11/22/2019, 2:05 AM
Hi @Thomas Adams, if I understand your question correctly, you'd like to store some credentials like username/password in environment variables; that's certainly doable with Prefect! https://docs.prefect.io/core/concepts/configuration.html#user-configuration should help you get started with environment variables. For an ETL example, https://docs.prefect.io/core/tutorials/etl.html#e-t-and-l should be helpful to get an understanding for how Prefect can be used to break down those tasks; adding a task to handle authorization as an upstream dependency (that returns a token or code) could be accomplished by adding a task which makes an http call to your auth provider. For that, I'd recommend the
requests
package or one of the various OAuth python libraries (https://oauth.net/code/python/ has a nice list!). Is that helpful?
t

Thomas Adams

11/22/2019, 2:08 AM
Okay maybe I need to spend more time with Prefect and overall proper coding. In my scripts I am a bad coder and rarely use functions to handle a task... I know I know.. Very helpful! I'll look at it more this weekend! Any youtube vidoes out there lol?
n

nicholas

11/22/2019, 2:11 AM
I should also add that Prefect has a type of Task to handle sensitive data in particular, the Secret Task. You can read more about that here: https://docs.prefect.io/api/unreleased/tasks/secrets.html#secret As for videos, which sort are you looking for? Unfortunately we don't have any Prefect-specific videos.
https://realpython.com/ is a great place for resources to learn a bit more about Python!
💯 2
t

Thomas Adams

11/22/2019, 2:13 AM
I went straight to numpy and pandas and did not learn how to properly use functions. Time to learn functions since that's what it looks like Prefect uses a lot.
Thanks for all the information guys.
😄 2
n

nicholas

11/22/2019, 2:17 AM
Prefect also has an Imperative API which you may find more comfortable than the functional API! You can read a bit more about that here: https://docs.prefect.io/core/concepts/flows.html#apis That may better match what you're used to coming from less functional packages!