Not sure if this is the right channel for submitting PRs But Prefect Community #prefect-contributors-archived

Not sure if this is the right channel for submitti...

Josh

01/15/2021, 10:09 PM

Not sure if this is the right channel for submitting PRs. But I have one to add chunking to GCS to enable handling of larger files or slower internet connections. Not sure how to best add tests for this in CI https://github.com/PrefectHQ/prefect/pull/3968

🙌 3

nicholas

01/15/2021, 10:24 PM

Amazing, thank you @Josh 🙏 ! The Core team will take a look at your PR when they can and will handle the review process through GitHub 🙂

Zanie

01/15/2021, 10:33 PM

I don’t manage the core repo, but I can point you towards the typical way to test something like this. Since the GCS library has its own tests and is out of our control, we just want to assert that if someone provides this new argument that it is properly passed to GCS. The typical way to do this is to “mock” the GCS function then assert that if

chunk_size

is given to one of these tasks that the object is created with that option set to the correct value.

Zanie

01/15/2021, 10:35 PM

An example of this is at

tests.tasks.gcp.test_gcs_upload_download.TestBlob

Zanie

01/15/2021, 10:36 PM

And at

tests.storage.test_gcs_storage.TestGCSStorage

Zanie

01/15/2021, 10:37 PM

Because the

blob

object we need to call assertions on is nested in several calls to the GCS library, we have to do several mocks e.g.

Copy code

blob_mock = MagicMock()
        bucket_mock = MagicMock(blob=MagicMock(return_value=blob_mock))
        google_client.return_value.get_bucket = MagicMock(return_value=bucket_mock)

Zanie

01/15/2021, 10:38 PM

The only one we’re really interested in is our

blob_mock

which we can assert was called with the

chunk_size

argument

4 Views

Open in Slack

Previous Next