matta
05/13/2020, 11:20 PMgspread
package, which has you do everything from method calls to an authentication object. So like you go gc = gspread.service_account(filename=<filename>)
and point it at a special credentials file, then everything is through that. Should I just pass it to Secrets? Is there any risk of sensitive credentials being cached somewhere if I define the gc
object within the Flow itself?nicholas
Jeremiah
matta
05/13/2020, 11:31 PMmatta
05/14/2020, 12:14 AMclass AuthenticateGsheets(SecretBase):
def __init__(self, credentials_filename: Union[str, pathlib.Path], **kwargs: Any):
self.credentials_filename = credentials_filename
super().__init__(**kwargs)
def run(self) -> gspread.client.Client:
return gspread.service_account(filename=self.credentials_filename)
Jeremiah
matta
05/16/2020, 1:10 AMmatta
05/16/2020, 1:11 AMJeremiah
matta
05/17/2020, 11:48 PMmatta
05/17/2020, 11:53 PM@task
def gsheet_macro(worsksheet: gspread.models.Worksheet, fn: Callable):
return Callable(worksheed)
gsheet_macro(gc, lambda x: x.find("Dough"))
matta
05/18/2020, 10:12 PMJeremiah
worksheet
object is Pickleable — but often, connection clients aren’tJeremiah
matta
05/18/2020, 11:30 PM@task
def gsheet_macro(
credentials_filename: Union[str, pathlib.Path] = None,
sheet_key: str = None,
worksheet_name: str = None,
fn: Callable,
):
client = AuthenticateGsheets(credentials_filename).run()
google_sheet = client.open_by_key(sheet_key)
worksheet = google_sheet.worksheet(worksheet_name)
return Callable(worksheet)
matta
05/18/2020, 11:31 PM.run()
inside another task? Does that mess with Prefect in some way?Jeremiah
@task
decorator at the top and instead make this just a function that returns tasksJeremiah
Jeremiah
@task
decorated function. In your current setup, where gsheet_macro
is decorated, fn
is going to be an input to your task (which seems unecessary). In addition, does AuthenticateGsheets
still need to be a task? maybe it can just be a helper function now?Jeremiah
def gsheet_helper(fn):
@task
def inner(credentials, sheet_key, name):
client = ...
return fn(...)
return inner
Jeremiah
Jeremiah
matta
05/18/2020, 11:50 PMAuthenticateGsheets
is a task right now to take advantage of subclassing SecretBase
as the "Prefectonic" way of handling authenticator objects. Not married to it if there's a better way!Jeremiah
.run()
should be totally fine!matta
05/19/2020, 12:07 AMmatta
05/19/2020, 12:18 AMWriteGsheetRow
and `ReadGsheeRow`` tasksmatta
05/19/2020, 12:18 AMmatta
05/19/2020, 12:19 AMJeremiah
Jeremiah
contrib/
folder, and this would be a great candidate - once it has full tests (from you, Prefect, or anyone in the community) it can graduate to the core!matta
05/19/2020, 12:47 AMmatta
05/19/2020, 12:47 AMmatta
05/19/2020, 12:48 AMLaura Lorenz (she/her)
05/19/2020, 1:40 AM/src/prefect/contrib/tasks/{optionally your package name}/google_sheets.py
(or whatever you called your module) — and if you want to go for tests you can look at https://github.com/PrefectHQ/prefect/blob/master/tests/tasks/gcp/test_gcs_upload_download.py for some inspiration, the tests in the TestsBuckets
and TestBlob
classes show how to use unittest’s MagicMock and pytest.monkeypatch so that your tests don’t actually have to communicate with the google sheets API!matta
05/19/2020, 1:54 AMmatta
05/19/2020, 1:54 AMmatta
05/20/2020, 2:21 AMnicholas