https://prefect.io logo
Title
n

Nelson Griffiths

12/05/2022, 7:33 PM
I am running into some errors today with pushing flows to my cloud storage bucket when applying deployments. I updated the most recent version of both
prefect
and
prefect-gcp
1
Found 1 deployment files.
Deploying from data_pipelines/ib_fees_availability/ib_fees_availability_flow.deployment.py
Traceback (most recent call last):
  File "/Users/nelsong/doubleriver/repos/data-pipelines/data_pipelines/ib_fees_availability/ib_fees_availability_flow.deployment.py", line 13, in <module>
    deployment = Deployment.build_from_flow(
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 205, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 156, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/nelsong/.pyenv/versions/3.10.5/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/nelsong/.pyenv/versions/3.10.5/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/prefect/deployments.py", line 732, in build_from_flow
    await deployment.upload_to_storage(ignore_file=ignore_file)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/prefect/deployments.py", line 580, in upload_to_storage
    file_count = await self.storage.put_directory(
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/prefect/filesystems.py", line 567, in put_directory
    return await self.filesystem.put_directory(
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/prefect/filesystems.py", line 356, in put_directory
    self.filesystem.put_file(f, fpath, overwrite=True)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/fsspec/asyn.py", line 113, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/fsspec/asyn.py", line 98, in sync
    raise return_result
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/gcsfs/core.py", line 1133, in _put_file
    await simple_upload(
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/gcsfs/core.py", line 1730, in simple_upload
    j = await fs._call(
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/gcsfs/core.py", line 422, in _call
    status, headers, info, contents = await self._request(
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/decorator.py", line 221, in fun
    return await caller(func, *(extras + args), **kw)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/gcsfs/retry.py", line 115, in retry_request
    return await func(*args, **kwargs)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/gcsfs/core.py", line 415, in _request
    validate_response(status, contents, path, args)
  File "/Users/nelsong/doubleriver/repos/data-pipelines/.venv/lib/python3.10/site-packages/gcsfs/retry.py", line 100, in validate_response
    raise ValueError("Bad Request: %s\n%s" % (path, msg))
ValueError: Bad Request: <https://storage.googleapis.com/upload/storage/v1/b/dr-dev-us-east1-prefect-flows/o>
Payload size invalid. Expected size: 2. Actual size: 3.
r

Ryan Peden

12/05/2022, 8:48 PM
I'm not sure 100% sure it is the cause here, but last time someone encountered this it was because their virtual environment was included in the deployment: https://linen.prefect.io/t/2180129/when-i-build-prefect-deployment-with-gcs-storage-it-tries-to Adding the venv to .prefect ignore seems to have resolved the issue.
n

Nelson Griffiths

12/05/2022, 8:52 PM
Thank you! I will give this a try and see if it fixes the error
That did it. Thank you!
r

Ryan Peden

12/05/2022, 8:54 PM
You're welcome! 😄