https://prefect.io logo
Title
j

Josh

01/15/2021, 10:09 PM
Not sure if this is the right channel for submitting PRs. But I have one to add chunking to GCS to enable handling of larger files or slower internet connections. Not sure how to best add tests for this in CI https://github.com/PrefectHQ/prefect/pull/3968
🙌 3
n

nicholas

01/15/2021, 10:24 PM
Amazing, thank you @Josh 🙏 ! The Core team will take a look at your PR when they can and will handle the review process through GitHub 🙂
z

Zanie

01/15/2021, 10:33 PM
I don’t manage the core repo, but I can point you towards the typical way to test something like this. Since the GCS library has its own tests and is out of our control, we just want to assert that if someone provides this new argument that it is properly passed to GCS. The typical way to do this is to “mock” the GCS function then assert that if
chunk_size
is given to one of these tasks that the object is created with that option set to the correct value.
An example of this is at
tests.tasks.gcp.test_gcs_upload_download.TestBlob
And at
tests.storage.test_gcs_storage.TestGCSStorage
Because the
blob
object we need to call assertions on is nested in several calls to the GCS library, we have to do several mocks e.g.
blob_mock = MagicMock()
        bucket_mock = MagicMock(blob=MagicMock(return_value=blob_mock))
        google_client.return_value.get_bucket = MagicMock(return_value=bucket_mock)
The only one we’re really interested in is our
blob_mock
which we can assert was called with the
chunk_size
argument