matta
05/13/2020, 11:20 PMgspread
package, which has you do everything from method calls to an authentication object. So like you go gc = gspread.service_account(filename=<filename>)
and point it at a special credentials file, then everything is through that. Should I just pass it to Secrets? Is there any risk of sensitive credentials being cached somewhere if I define the gc
object within the Flow itself?nicholas
05/13/2020, 11:24 PMJeremiah
05/13/2020, 11:29 PMmatta
05/13/2020, 11:31 PMclass AuthenticateGsheets(SecretBase):
def __init__(self, credentials_filename: Union[str, pathlib.Path], **kwargs: Any):
self.credentials_filename = credentials_filename
super().__init__(**kwargs)
def run(self) -> gspread.client.Client:
return gspread.service_account(filename=self.credentials_filename)
Jeremiah
05/14/2020, 11:35 AMmatta
05/16/2020, 1:10 AMJeremiah
05/16/2020, 1:55 AMmatta
05/17/2020, 11:48 PM@task
def gsheet_macro(worsksheet: gspread.models.Worksheet, fn: Callable):
return Callable(worksheed)
gsheet_macro(gc, lambda x: x.find("Dough"))
Jeremiah
05/18/2020, 11:25 PMworksheet
object is Pickleable — but often, connection clients aren’tmatta
05/18/2020, 11:30 PM@task
def gsheet_macro(
credentials_filename: Union[str, pathlib.Path] = None,
sheet_key: str = None,
worksheet_name: str = None,
fn: Callable,
):
client = AuthenticateGsheets(credentials_filename).run()
google_sheet = client.open_by_key(sheet_key)
worksheet = google_sheet.worksheet(worksheet_name)
return Callable(worksheet)
.run()
inside another task? Does that mess with Prefect in some way?Jeremiah
05/18/2020, 11:42 PM@task
decorator at the top and instead make this just a function that returns tasks@task
decorated function. In your current setup, where gsheet_macro
is decorated, fn
is going to be an input to your task (which seems unecessary). In addition, does AuthenticateGsheets
still need to be a task? maybe it can just be a helper function now?def gsheet_helper(fn):
@task
def inner(credentials, sheet_key, name):
client = ...
return fn(...)
return inner
matta
05/18/2020, 11:50 PMAuthenticateGsheets
is a task right now to take advantage of subclassing SecretBase
as the "Prefectonic" way of handling authenticator objects. Not married to it if there's a better way!Jeremiah
05/19/2020, 12:05 AM.run()
should be totally fine!matta
05/19/2020, 12:07 AMWriteGsheetRow
and `ReadGsheeRow`` tasksJeremiah
05/19/2020, 12:41 AMcontrib/
folder, and this would be a great candidate - once it has full tests (from you, Prefect, or anyone in the community) it can graduate to the core!matta
05/19/2020, 12:47 AMLaura Lorenz (she/her)
05/19/2020, 1:40 AM/src/prefect/contrib/tasks/{optionally your package name}/google_sheets.py
(or whatever you called your module) — and if you want to go for tests you can look at https://github.com/PrefectHQ/prefect/blob/master/tests/tasks/gcp/test_gcs_upload_download.py for some inspiration, the tests in the TestsBuckets
and TestBlob
classes show how to use unittest’s MagicMock and pytest.monkeypatch so that your tests don’t actually have to communicate with the google sheets API!matta
05/19/2020, 1:54 AMnicholas
05/20/2020, 2:26 AM