Ilya Galperin
08/23/2022, 2:40 PMprefect cloud login -k $PREFECT_API_KEY -w $PREFECT_WORKSPACE
in order to run a block.save
Python method? If we try authenticating through simply setting our default profile and running prefect config set PREFECT_API_KEY/PREFECT_API_URL
but not explicitly calling prefect cloud, we get an error when trying to save a storage block (noted in replies). Is there a recommended pattern to use for CI/CD to a save a block otherwise?Traceback (most recent call last):
File "/home/runner/work/gha-prefect/gha-prefect/flows/flow_a/store_flow.py", line 25, in <module>
storage_block = create_storage_block(bucket, flow)
File "/home/runner/work/gha-prefect/gha-prefect/flows/flow_a/store_flow.py", line 16, in create_storage_block
block.save(block_name, overwrite=True)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/blocks/core.py", line 724, in save
document_id = await self._save(name=name, overwrite=overwrite)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/blocks/core.py", line 677, in _save
await self.register_type_and_schema(client=client)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/blocks/core.py", line 619, in register_type_and_schema
block_type = await client.read_block_type_by_slug(
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/client.py", line 1120, in read_block_type_by_slug
return BlockType.parse_obj(response.json())
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/httpx/_models.py", line 743, in json
return jsonlib.loads(self.text, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Error: Process completed with exit code 1.
Khuyen Tran
08/23/2022, 3:26 PMIlya Galperin
08/23/2022, 4:29 PMKhuyen Tran
08/23/2022, 5:00 PMIlya Galperin
08/23/2022, 5:02 PMKhuyen Tran
08/23/2022, 5:18 PMIlya Galperin
08/23/2022, 5:22 PMprefect profile use default
before prefect cloud login
to avoid the prompt.Khuyen Tran
08/23/2022, 6:22 PMprefect profile use xyz
prefect config set PREFECT_API_KEY="xxx"
prefect config set PREFECT_API_KEY="yyy"
Ilya Galperin
08/23/2022, 6:29 PMKhuyen Tran
08/23/2022, 6:32 PMIlya Galperin
08/23/2022, 8:03 PMKhuyen Tran
08/23/2022, 8:14 PMprefect profile delete <name-of-your-profile>
Then run prefect cloud login -k API-KEY
Ilya Galperin
08/23/2022, 8:25 PMprefect config set PREFECT_API_KEY/PREFECT_API_URL
and running the below script is generating the same error as described before.
from prefect.filesystems import S3
def create_storage_block(bucket, flow):
block_name = f"{flow}-code"
aws_access_key_id = "xyz123"
aws_secret_access_key = "xyz123"
block = S3(
bucket_path=f"{bucket}/flows/{flow}",
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
)
block.save(block_name, overwrite=True)
return block_name
if __name__ == "__main__":
bucket = "mybucket"
flow = "myflow"
storage_block = create_storage_block(bucket, flow)
prefect cloud login
, the script works.Khuyen Tran
08/23/2022, 8:30 PMprefect config view
Just to double checkIlya Galperin
08/23/2022, 8:44 PMPREFECT_PROFILE='xyz2'
PREFECT_API_KEY='my_api_key' (from profile)
PREFECT_API_URL='<https://app.prefect.cloud/accounts/xyz/workspaces/xyz>' (from profile)
prefect cloud login -k my_key
afterwards it’s switching profile back to default from xyz2prefect cloud login -k my_key 01:43:54 PM
Logged in to Prefect Cloud using profile 'default'.
Workspace is currently set to 'account/workspace'. The workspace can be changed using `prefect cloud workspace set`.
Tim-Oliver
08/24/2022, 1:44 PMLocalFileSystem
block in the cloud but get the same `JSONDecodeError`when I try to use the block locally. Did you find a solution for this already?Ilya Galperin
08/24/2022, 3:28 PM