I am trying to run a deployment via Docker However I receive

Question

I am trying to run a deployment via Docker However I receive the following error ```FileNotFoundError Errno 2 No such file or directory opt prefect flows ``` Which is surprising because I don t use the above location to reference the flow This is what my Docker file look like ```FROM python 3 8 slim buster ARG PREFECT API KEY ENV PREFECT API KEY=$PREFECT API KEY ARG PREFECT API URL ENV PREFECT API URL=$PREFECT API URL ARG GCP DATASET NAME ENV GCP DATASET NAME=$GCP DATASET NAME ARG GCP DATASET TABLE NAME ENV GCP DATASET TABLE NAME=$GCP DATASET TABLE NAME ARG GCP PROJECT ID ENV GCP PROJECT ID=$GCP PROJECT ID ARG GCP REGION ENV GCP REGION=$GCP REGION COPY poetry lock COPY pyproject toml RUN pip install poetry trusted host <http pypi python org|pypi python org> no cache dir RUN poetry config virtualenvs create false RUN poetry install no root without dev RUN mkdir scripts copy scripts scripts RUN mkdir config COPY config config RUN mkdir p dbt xetra COPY dbt xetra dbt xetra``` Any idea why Prefect is looking for the flow in the opt prefect flows directory I am running this via the Cloud

alex · Answer

Hey < sundeep> Are you using an infrastructure block or a work pool for your deployment

Robert Banick · Answer

< sundeep> I ve had similar issues setting up my stack using flows over AWS with a Github infra block Not sure if it s relevant to a Docker deployment but As far as I can tell prefect checks my block into ` opt prefect ` and then auto looks for the ` opt prefect flows` Specifying `path= lt flow dir gt ` on my deployments was enough for prefect to find them Prefect folks reading please note that I didn t find the documentation at all on how and where Prefect manages directories inside containers

sundeep · Answer

< alex> I am using an infrastructure block

sundeep · Answer

< Robert Banick> Let me try that Thank you

sundeep · Answer

< Robert Banick> that was indeed the issue Thanks However I have run into another error Since my flow references the GCP Credentials block inside the container this piece of code throws the following error ``` task log prints=True name= list files from gcs def list gcs credentials bucket name str prefix str delimiter str gt Iterator str Loop through files in GCS directory storage client = storage Client credentials=credentials project=credentials project id Note Client list blobs requires at least package version 1 17 0 blobs = storage client list blobs bucket name prefix=prefix delimiter=delimiter for blob in blobs yield blob name```

sundeep · Answer

Error ```09 34 43 973 | INFO | Flow run burrowing parakeet gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 09 34 43 910 | ERROR | Flow run attentive basilisk Encountered exception during execution 09 34 44 418 | ERROR | Flow run attentive basilisk Finished in state Failed Flow run encountered an exception gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 n ```

sundeep · Answer

I don t see this issue when I run the flow locally Do I need to copy the credentials file inside the Docker I am just wary of this as the image is published to the hub The example provided here doesn t copy any credentials to the Docker image <https github com discdiver prefect zoomcamp> Any idea how to resolve this Appreciate your help

Robert Banick · Answer

Looks like a GCFS issue sorry I don t have any experience with this

sundeep · Answer

No worries Thanks

sundeep · Answer

< alex> do you have idea how to resolve this

alex · Answer

You can use a `GcpCredentials` block to retrieve credentials in your flow Here s an example of how to create those blocks <https github com discdiver prefect zoomcamp blob main blocks make gcp blocks py>

sundeep · Answer

< alex> i have a gcp credentials block and also a json block to retrieve all the relevant credentials and env variables but it doesn t help

sundeep · Answer

It fails while reading the parquet df ``` task def gcs to bq file str credentials bucket name str gcp dataset name str gcp dataset table str gt None Loading data to BigQuery script runs but no data in BigQuery setting google client client = bigquery Client credentials=credentials project=credentials project id try df = pd read parquet f gs bucket name file df key = df ISIN fillna + + df Date astype str fillna + + df Time astype str fillna df key = df key str lower str encode utf 8 apply sha print f Number of rows df shape table schema= name key type STRING name ISIN type STRING name Mnemonic type STRING name SecurityDesc type STRING name SecurityType type STRING name Currency type STRING name SecurityID type INT64 name Date type TIMESTAMP name Time type STRING name StartPrice type FLOAT64 name MaxPrice type FLOAT64 name MinPrice type FLOAT64 name EndPrice type FLOAT64 name TradedVolume type INT64 name NumberOfTrades type INT64 job config = bigquery LoadJobConfig job = client load table from dataframe df f gcp dataset name gcp dataset table job config=job config job result except FileNotFoundError pass except pyarrow lib ArrowTypeError pass```

sundeep · Answer

Here is the stack trace ```00 21 16 666 | INFO | prefect agent Submitting flow run c267fe95 6940 4346 9e2d f1791df6a3bd 00 21 18 360 | INFO | prefect infrastructure docker container Pulling image sl02 xetra latest 00 21 21 409 | INFO | prefect infrastructure docker container Creating Docker container nickel crow 00 21 21 454 | INFO | prefect infrastructure docker container Docker container nickel crow has status created 00 21 21 746 | INFO | prefect infrastructure docker container Docker container nickel crow has status running 00 21 22 039 | INFO | prefect agent Completed submission of flow run c267fe95 6940 4346 9e2d f1791df6a3bd usr local lib python3 8 runpy py 127 RuntimeWarning prefect engine found in sys modules after import of package prefect but prior to execution of prefect engine this may result in unpredictable behaviour warn RuntimeWarning msg 18 51 28 193 | INFO | Flow run nickel crow Downloading flow code from storage at scripts 18 51 31 673 | INFO | Flow run nickel crow Created task run dataset load check 0 for task dataset load check 18 51 31 675 | INFO | Flow run nickel crow Executing dataset load check 0 immediately 18 51 33 414 | INFO | Task run dataset load check 0 Finished in state Completed 18 51 34 990 | INFO | Flow run nickel crow Created subflow run cordial elephant for flow Pipeline to read files from GCS and load to BigQuery 18 51 36 587 | INFO | Flow run cordial elephant Created task run list files from gcs 0 for task list files from gcs 18 51 36 588 | INFO | Flow run cordial elephant Executing list files from gcs 0 immediately 18 51 37 444 | INFO | Task run list files from gcs 0 list gcs lt google oauth2 service account Credentials object at 0x7fb9794def70 gt 18 51 38 392 | INFO | Task run list files from gcs 0 Finished in state Completed 18 51 38 394 | INFO | Flow run cordial elephant file data xetra 2022 04 05 2022 04 05 BINS XETR07 parquet 18 51 38 742 | INFO | Flow run cordial elephant Created task run gcs to bq 0 for task gcs to bq 18 51 38 743 | INFO | Flow run cordial elephant Executing gcs to bq 0 immediately 18 51 39 464 | INFO | Task run gcs to bq 0 Inside gcs to bq xetra ds data xetra 2022 04 05 2022 04 05 BINS XETR07 parquet xetra ds xetra stocks 18 51 42 668 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 1 of 3 Reason timed out 18 51 42 670 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 2 of 3 Reason Errno 111 Connection refused 18 51 42 672 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 3 of 3 Reason Errno 111 Connection refused 18 51 42 673 | WARNING | google auth default Authentication failed using Compute Engine authentication due to unavailable metadata server 18 51 42 680 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 1 of 5 Reason HTTPConnectionPool host= metadata google internal port=80 Max retries exceeded with url computeMetadata v1 instance service accounts default recursive=true Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fb9787e3640 gt Failed to establish a new connection Errno 2 Name or service not known 18 51 42 686 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 2 of 5 Reason HTTPConnectionPool host= metadata google internal port=80 Max retries exceeded with url computeMetadata v1 instance service accounts default recursive=true Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fb9787e3ac0 gt Failed to establish a new connection Errno 2 Name or service not known 18 51 42 691 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 3 of 5 Reason HTTPConnectionPool host= metadata google internal port=80 Max retries exceeded with url computeMetadata v1 instance service accounts default recursive=true Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fb9787e3d90 gt Failed to establish a new connection Errno 2 Name or service not known 18 51 42 696 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 4 of 5 Reason HTTPConnectionPool host= metadata google internal port=80 Max retries exceeded with url computeMetadata v1 instance service accounts default recursive=true Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fb9787e5790 gt Failed to establish a new connection Errno 2 Name or service not known 18 51 42 701 | WARNING | google auth compute engine metadata Compute Engine Metadata server unavailable on attempt 5 of 5 Reason HTTPConnectionPool host= metadata google internal port=80 Max retries exceeded with url computeMetadata v1 instance service accounts default recursive=true Caused by ```

sundeep · Answer

```NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fb9787e5670 gt Failed to establish a new connection Errno 2 Name or service not known 18 51 43 083 | INFO | Task run gcs to bq 0 Traceback most recent call last 18 51 43 084 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 114 in retry request return await func args kwargs 18 51 43 086 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 411 in request validate response status contents path args 18 51 43 088 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 101 in validate response raise HttpError error 18 51 43 089 | INFO | Task run gcs to bq 0 gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 18 51 43 082 | ERROR | gcsfs request non retriable exception Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 18 51 43 094 | INFO | Task run gcs to bq 0 Traceback most recent call last 18 51 43 095 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages prefect engine py line 1551 in orchestrate task run result = await call aresult 18 51 43 096 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future 18 51 43 097 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs 18 51 43 098 | INFO | Task run gcs to bq 0 File scripts gcs to bq py line 84 in gcs to bq df = pd read parquet f gs bucket name file 18 51 43 099 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pandas io parquet py line 509 in read parquet return impl read 18 51 43 100 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pandas io parquet py line 227 in read pa table = self api parquet read table 18 51 43 101 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pyarrow parquet core py line 2926 in read table dataset = ParquetDatasetV2 18 51 43 102 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pyarrow parquet core py line 2452 in init finfo = filesystem get file info path or paths 18 51 43 103 | INFO | Task run gcs to bq 0 File pyarrow fs pyx line 571 in pyarrow fs FileSystem get file info 18 51 43 104 | INFO | Task run gcs to bq 0 File pyarrow error pxi line 144 in pyarrow lib pyarrow internal check status 18 51 43 104 | INFO | Task run gcs to bq 0 File pyarrow fs pyx line 1490 in pyarrow fs cb get file info 18 51 43 105 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pyarrow fs py line 330 in get file info info = <http self fs info|self fs info> path 18 51 43 106 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages fsspec asyn py line 114 in wrapper return sync self loop func args kwargs 18 51 43 107 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages fsspec asyn py line 99 in sync raise return result 18 51 43 108 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages fsspec asyn py line 54 in runner result 0 = await coro 18 51 43 109 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 791 in info exact = await self get object path 18 51 43 110 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 491 in get object res = await self call 18 51 43 111 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 418 in call status headers info contents = await self request 18 51 43 111 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages decorator py line 221 in fun return await caller func extras + args kw ```

sundeep · Answer

```18 51 43 112 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 149 in retry request raise e 18 51 43 113 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 114 in retry request return await func args kwargs 18 51 43 114 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 411 in request validate response status contents path args 18 51 43 115 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 101 in validate response raise HttpError error 18 51 43 116 | INFO | Task run gcs to bq 0 gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 18 51 43 117 | INFO | Task run gcs to bq 0 Traceback most recent call last 18 51 43 118 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages prefect engine py line 1551 in orchestrate task run result = await call aresult 18 51 43 119 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future 18 51 43 120 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs 18 51 43 121 | INFO | Task run gcs to bq 0 File scripts gcs to bq py line 84 in gcs to bq df = pd read parquet f gs bucket name file 18 51 43 122 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pandas io parquet py line 509 in read parquet return impl read 18 51 43 123 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pandas io parquet py line 227 in read pa table = self api parquet read table 18 51 43 124 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pyarrow parquet core py line 2926 in read table dataset = ParquetDatasetV2 18 51 43 125 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pyarrow parquet core py line 2452 in init finfo = filesystem get file info path or paths 18 51 43 126 | INFO | Task run gcs to bq 0 File pyarrow fs pyx line 571 in pyarrow fs FileSystem get file info 18 51 43 127 | INFO | Task run gcs to bq 0 File pyarrow error pxi line 144 in pyarrow lib pyarrow internal check status 18 51 43 127 | INFO | Task run gcs to bq 0 File pyarrow fs pyx line 1490 in pyarrow fs cb get file info 18 51 43 128 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages pyarrow fs py line 330 in get file info info = <http self fs info|self fs info> path 18 51 43 129 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages fsspec asyn py line 114 in wrapper return sync self loop func args kwargs 18 51 43 130 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages fsspec asyn py line 99 in sync raise return result 18 51 43 130 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages fsspec asyn py line 54 in runner result 0 = await coro 18 51 43 131 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 791 in info exact = await self get object path 18 51 43 132 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 491 in get object res = await self call 18 51 43 133 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 418 in call status headers info contents = await self request 18 51 43 133 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages decorator py line 221 in fun return await caller func extras + args kw 18 51 43 134 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 149 in retry request raise e 18 51 43 135 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 114 in retry request return await func args kwargs 18 51 43 136 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs core py line 411 in request validate response status contents path args 18 51 43 136 | INFO | Task run gcs to bq 0 File usr local lib python3 8 site packages gcsfs retry py line 101 in validate response raise HttpError error 18 51 43 137 | INFO | Task run gcs to bq 0 gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 18 51 43 092 | ERROR | Task run gcs to bq 0 Encountered exception during execution 18 51 43 545 | ERROR | Task run gcs to bq 0 Finished in state Failed Task run encountered an exception gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 n 18 51 43 549 | INFO | Flow run nickel crow Traceback most recent call last 18 51 43 550 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 669 in orchestrate flow run result = await flow call aresult 18 51 43 551 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future 18 51 43 552 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs 18 51 43 553 | INFO | Flow run nickel crow File scripts gcs to bq py line 265 in pipeline gcs to bq file credentials bucket name gcp dataset name gcp dataset table 18 51 43 554 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect tasks py line 485 in call return enter task run engine 18 51 43 555 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 972 in enter task run engine return from sync wait for call in loop thread begin run 18 51 43 556 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency api py line 137 in wait for call in loop thread return call result 18 51 43 556 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 173 in result return self future result timeout=timeout 18 51 43 557 | INFO | Flow run nickel crow File usr local lib python3 8 concurrent futures base py line 437 in result return self get result 18 51 43 558 | INFO | Flow run nickel crow File usr local lib python3 8 concurrent futures base py line 389 in get result raise self exception 18 51 43 559 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 218 in run async result = await coro 18 51 43 560 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 1137 in get task call return value return await future result 18 51 43 561 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect futures py line 241 in result return await final state result raise on failure=raise on failure fetch=True 18 51 43 562 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect states py line 91 in get state result raise await get state exception state 18 51 43 563 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 1551 in orchestrate task run result = await call aresult 18 51 43 564 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future 18 51 43 565 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs 18 51 43 566 | INFO | Flow run nickel crow File scripts gcs to bq py line 84 in gcs to bq df = pd read parquet f gs bucket name file 18 51 43 566 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pandas io parquet py line 509 in read parquet return impl read 18 51 43 567 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pandas io parquet py line 227 in read pa table = self api parquet read table 18 51 43 568 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pyarrow parquet core py line 2926 in read table dataset = ParquetDatasetV2 18 51 43 569 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pyarrow parquet core py line 2452 in init finfo = filesystem get file info path or paths 18 51 43 570 | INFO | Flow run nickel crow File pyarrow fs pyx line 571 in pyarrow fs FileSystem get file info 18 51 43 571 | INFO | Flow run nickel crow File pyarrow error pxi line 144 in pyarrow lib pyarrow internal check status 18 51 43 572 | INFO | Flow run nickel crow File pyarrow fs pyx line 1490 in pyarrow fs cb get file info```

sundeep · Answer

```18 51 43 572 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pyarrow fs py line 330 in get file info info = <http self fs info|self fs info> path 18 51 43 573 | INFO | Flow run nickel crow File usr local lib python3 8 site packages fsspec asyn py line 114 in wrapper return sync self loop func args kwargs 18 51 43 574 | INFO | Flow run nickel crow File usr local lib python3 8 site packages fsspec asyn py line 99 in sync raise return result 18 51 43 575 | INFO | Flow run nickel crow File usr local lib python3 8 site packages fsspec asyn py line 54 in runner result 0 = await coro 18 51 43 576 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 791 in info exact = await self get object path 18 51 43 577 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 491 in get object res = await self call 18 51 43 578 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 418 in call status headers info contents = await self request 18 51 43 578 | INFO | Flow run nickel crow File usr local lib python3 8 site packages decorator py line 221 in fun return await caller func extras + args kw 18 51 43 579 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs retry py line 149 in retry request raise e 18 51 43 580 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs retry py line 114 in retry request return await func args kwargs 18 51 43 581 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 411 in request validate response status contents path args 18 51 43 582 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs retry py line 101 in validate response raise HttpError error 18 51 43 583 | INFO | Flow run nickel crow gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 18 51 43 585 | INFO | Flow run nickel crow Traceback most recent call last 18 51 43 586 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 669 in orchestrate flow run result = await flow call aresult 18 51 43 587 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future 18 51 43 588 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs 18 51 43 589 | INFO | Flow run nickel crow File scripts gcs to bq py line 265 in pipeline gcs to bq file credentials bucket name gcp dataset name gcp dataset table 18 51 43 590 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect tasks py line 485 in call return enter task run engine 18 51 43 590 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 972 in enter task run engine return from sync wait for call in loop thread begin run ```

sundeep · Answer

```18 51 43 591 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency api py line 137 in wait for call in loop thread return call result 18 51 43 592 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 173 in result return self future result timeout=timeout 18 51 43 593 | INFO | Flow run nickel crow File usr local lib python3 8 concurrent futures base py line 437 in result return self get result 18 51 43 593 | INFO | Flow run nickel crow File usr local lib python3 8 concurrent futures base py line 389 in get result raise self exception 18 51 43 594 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 218 in run async result = await coro 18 51 43 595 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 1137 in get task call return value return await future result 18 51 43 596 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect futures py line 241 in result return await final state result raise on failure=raise on failure fetch=True 18 51 43 597 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect states py line 91 in get state result raise await get state exception state 18 51 43 597 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect engine py line 1551 in orchestrate task run result = await call aresult 18 51 43 598 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future 18 51 43 599 | INFO | Flow run nickel crow File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs 18 51 43 600 | INFO | Flow run nickel crow File scripts gcs to bq py line 84 in gcs to bq df = pd read parquet f gs bucket name file 18 51 43 600 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pandas io parquet py line 509 in read parquet return impl read 18 51 43 601 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pandas io parquet py line 227 in read pa table = self api parquet read table 18 51 43 602 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pyarrow parquet core py line 2926 in read table dataset = ParquetDatasetV2 18 51 43 603 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pyarrow parquet core py line 2452 in init finfo = filesystem get file info path or paths 18 51 43 604 | INFO | Flow run nickel crow File pyarrow fs pyx line 571 in pyarrow fs FileSystem get file info 18 51 43 604 | INFO | Flow run nickel crow File pyarrow error pxi line 144 in pyarrow lib pyarrow internal check status 18 51 43 605 | INFO | Flow run nickel crow File pyarrow fs pyx line 1490 in pyarrow fs cb get file info 18 51 43 606 | INFO | Flow run nickel crow File usr local lib python3 8 site packages pyarrow fs py line 330 in get file info info = <http self fs info|self fs info> path 18 51 43 606 | INFO | Flow run nickel crow File usr local lib python3 8 site packages fsspec asyn py line 114 in wrapper return sync self loop func args kwargs 18 51 43 607 | INFO | Flow run nickel crow File usr local lib python3 8 site packages fsspec asyn py line 99 in sync raise return result 18 51 43 608 | INFO | Flow run nickel crow File usr local lib python3 8 site packages fsspec asyn py line 54 in runner result 0 = await coro 18 51 43 609 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 791 in info exact = await self get object path 18 51 43 609 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 491 in get object res = await self call 18 51 43 610 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 418 in call status headers info contents = await self request 18 51 43 611 | INFO | Flow run nickel crow File usr local lib python3 8 site packages decorator py line 221 in fun return await caller func extras + args kw 18 51 43 612 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs retry py line 149 in retry request raise e 18 51 43 612 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs retry py line 114 in retry request return await func args kwargs 18 51 43 613 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs core py line 411 in request validate response status contents path args 18 51 43 614 | INFO | Flow run nickel crow File usr local lib python3 8 site packages gcsfs retry py line 101 in validate response raise HttpError error ```

sundeep · Answer

```18 51 43 615 | INFO | Flow run nickel crow gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 18 51 43 548 | ERROR | Flow run cordial elephant Encountered exception during execution 18 51 44 073 | ERROR | Flow run cordial elephant Finished in state Failed Flow run encountered an exception gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 n 18 51 44 075 | ERROR | Flow run nickel crow Encountered exception during execution Traceback most recent call last File usr local lib python3 8 site packages prefect engine py line 669 in orchestrate flow run result = await flow call aresult File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs File scripts gcs to bq py line 305 in main num = pipeline bucket name prefix gcp dataset name gcp dataset table File usr local lib python3 8 site packages prefect flows py line 468 in call return enter flow run engine from flow call File usr local lib python3 8 site packages prefect engine py line 184 in enter flow run engine from flow call retval = from sync wait for call in loop thread File usr local lib python3 8 site packages prefect internal concurrency api py line 137 in wait for call in loop thread return call result File usr local lib python3 8 site packages prefect internal concurrency calls py line 173 in result return self future result timeout=timeout File usr local lib python3 8 concurrent futures base py line 437 in result return self get result File usr local lib python3 8 concurrent futures base py line 389 in get result raise self exception File usr local lib python3 8 site packages prefect internal concurrency calls py line 218 in run async result = await coro File usr local lib python3 8 site packages prefect client utilities py line 40 in with injected client return await fn args kwargs File usr local lib python3 8 site packages prefect engine py line 566 in create and begin subflow run return await terminal state result fetch=True File usr local lib python3 8 site packages prefect states py line 91 in get state result raise await get state exception state File usr local lib python3 8 site packages prefect engine py line 669 in orchestrate flow run result = await flow call aresult File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs File scripts gcs to bq py line 265 in pipeline gcs to bq file credentials bucket name gcp dataset name gcp dataset table File usr local lib python3 8 site packages prefect tasks py line 485 in call return enter task run engine File usr local lib python3 8 site packages prefect engine py line 972 in enter task run engine return from sync wait for call in loop thread begin run File usr local lib python3 8 site packages prefect internal concurrency api py line 137 in wait for call in loop thread return call result File usr local lib python3 8 site packages prefect internal concurrency calls py line 173 in result return self future result timeout=timeout File usr local lib python3 8 concurrent futures base py line 437 in result return self get result File usr local lib python3 8 concurrent futures base py line 389 in get result raise self exception File usr local lib python3 8 site packages prefect internal concurrency calls py line 218 in run async result = await coro File usr local lib python3 8 site packages prefect engine py line 1137 in get task call return value return await future result File usr local lib python3 8 site packages prefect futures py line 241 in result return await final state result raise on failure=raise on failure fetch=True File usr local lib python3 8 site packages prefect states py line 91 in get state result raise await get state exception state File usr local lib python3 8 site packages prefect engine py line 1551 in orchestrate task run result = await call aresult File usr local lib python3 8 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future File usr local lib python3 8 site packages prefect internal concurrency calls py line 194 in run sync result = self fn self args self kwargs File scripts gcs to bq py line 84 in gcs to bq df = pd read parquet f gs bucket name file File usr local lib python3 8 site packages pandas io parquet py line 509 in read parquet return impl read File usr local lib python3 8 site packages pandas io parquet py line 227 in read pa table = self api parquet read table File usr local lib python3 8 site packages pyarrow parquet core py line 2926 in read table dataset = ParquetDatasetV2 File usr local lib python3 8 site packages pyarrow parquet core py line 2452 in init finfo = filesystem get file info path or paths File pyarrow fs pyx line 571 in pyarrow fs FileSystem get file info File pyarrow error pxi line 144 in pyarrow lib pyarrow internal check status File pyarrow fs pyx line 1490 in pyarrow fs cb get file info File usr local lib python3 8 site packages pyarrow fs py line 330 in get file info info = <http self fs info|self fs info> path File usr local lib python3 8 site packages fsspec asyn py line 114 in wrapper return sync self loop func args kwargs File usr local lib python3 8 site packages fsspec asyn py line 99 in sync raise return result File usr local lib python3 8 site packages fsspec asyn py line 54 in runner result 0 = await coro File usr local lib python3 8 site packages gcsfs core py line 791 in info exact = await self get object path File usr local lib python3 8 site packages gcsfs core py line 491 in get object res = await self call File usr local lib python3 8 site packages gcsfs core py line 418 in call status headers info contents = await self request File usr local lib python3 8 site packages decorator py line 221 in fun return await caller func extras + args kw File usr local lib python3 8 site packages gcsfs retry py line 149 in retry request raise e File usr local lib python3 8 site packages gcsfs retry py line 114 in retry request return await func args kwargs File usr local lib python3 8 site packages gcsfs core py line 411 in request validate response status contents path args File usr local lib python3 8 site packages gcsfs retry py line 101 in validate response raise HttpError error gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 18 51 44 571 | ERROR | Flow run nickel crow Finished in state Failed Flow run encountered an exception gcsfs retry HttpError Anonymous caller does not have storage objects get access to the Google Cloud Storage object Permission storage objects get denied on resource or it may not exist 401 n ```

kasteph · Answer

< sundeep> your service account or principal is missing the `storage objects get` permission for that bucket