https://prefect.io logo
#prefect-community
Title
# prefect-community
h

haris khan

08/04/2022, 10:44 AM
when i build prefect deployment with gcs storage it tries to copy all my files including the virtaul enviroment to the gcs bucket and then fails (venv) haris@haris-Inspiron-3593:~/Desktop/snowflake$ prefect deployment build workflow/venue_flow.py:gen_venue_flow -n genvenueflow_deployment -t n1-standard-4 -i kubernetes-job --storage-block gcs/prefect2-snowflake /home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/flows.py176 UserWarning: A flow named 'gen_venue' and defined at 'workflow/venue_flow.py:6' conflicts with another flow. Consider specifying a unique
name
parameter in the flow definition:
@flow(name='my_unique_name', ...)
warnings.warn( Found flow 'gen_venue' Manifest created at '/home/haris/Desktop/snowflake/gen_venue_flow-manifest.json'. Traceback (most recent call last): File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper return fn(*args, **kwargs) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper return run_async_in_new_loop(async_fn, *args, **kwargs) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop return anyio.run(partial(__fn, *args, **kwargs)) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 56, in run return asynclib.run(func, *args, **backend_options) # type: ignore File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 233, in run return native_run(wrapper(), debug=debug) File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper return await func(*args) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/cli/deployment.py", line 502, in build file_count = await storage.put_directory() File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/filesystems.py", line 428, in put_directory return await self.filesystem.put_directory( File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/filesystems.py", line 250, in put_directory self.filesystem.put_file(f, fpath) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 86, in wrapper return sync(self.loop, func, *args, **kwargs) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 66, in sync raise return_result File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 26, in _runner result[0] = await coro File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 1045, in _put_file await simple_upload( File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 1596, in simple_upload j = await fs._call( File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 392, in _call status, headers, info, contents = await self._request( File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/decorator.py", line 221, in fun return await caller(func, *(extras + args), **kw) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/retry.py", line 115, in retry_request return await func(*args, **kwargs) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 384, in _request validate_response(status, contents, path, args) File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/retry.py", line 100, in validate_response raise ValueError("Bad Request: %s\n%s" % (path, msg)) ValueError: Bad Request: https://storage.googleapis.com/upload/storage/v1/b/prefect2-snowflake/o Payload size invalid. Expected size: 2. Actual size: 3. An exception occurred.
is there a way to exclude the virtual enviroment ?
solved it by moving my virtual enviroment folder outside the directory but there is another issue everytime i run the deployment build command with storage it overwrites all files again and not just the updated one
ps : ive got multiple deployments
k

Khuyen Tran

08/04/2022, 6:08 PM
Yes, you can do that with
.prefectignore
. It is similar to
.gitignore
For example, if you want to exclude a virtual environment named
venv
, add the following to your `.prefectignore`:
Copy code
venv
11 Views