https://prefect.io logo
Powered by
Title
i

Ilya Galperin

08/23/2022, 2:40 PM
Do we need to authenticate with prefect cloud via 
prefect cloud login -k $PREFECT_API_KEY -w $PREFECT_WORKSPACE
in order to run a 
block.save
Python method? If we try authenticating through simply setting our default profile and running 
prefect config set PREFECT_API_KEY/PREFECT_API_URL
but not explicitly calling prefect cloud, we get an error when trying to save a storage block (noted in replies). Is there a recommended pattern to use for CI/CD to a save a block otherwise?
Traceback (most recent call last):
  File "/home/runner/work/gha-prefect/gha-prefect/flows/flow_a/store_flow.py", line 25, in <module>
    storage_block = create_storage_block(bucket, flow)
  File "/home/runner/work/gha-prefect/gha-prefect/flows/flow_a/store_flow.py", line 16, in create_storage_block
    block.save(block_name, overwrite=True)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/blocks/core.py", line 724, in save
    document_id = await self._save(name=name, overwrite=overwrite)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/blocks/core.py", line 677, in _save
    await self.register_type_and_schema(client=client)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/blocks/core.py", line 619, in register_type_and_schema
    block_type = await client.read_block_type_by_slug(
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/prefect/client.py", line 1120, in read_block_type_by_slug
    return BlockType.parse_obj(response.json())
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/httpx/_models.py", line 743, in json
    return jsonlib.loads(self.text, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Error: Process completed with exit code 1.
k

Khuyen Tran

08/23/2022, 3:26 PM
Here is an example of how you can use Prefect for CI/CD. Can you try that to see if it works?
i

Ilya Galperin

08/23/2022, 4:29 PM
Hi Khuyen — yes, we’re doing the same steps from the Create blocks job here and receiving the above error and receiving the error above.
If we add prefect cloud login after line 53 this works but otherwise we get an error.
k

Khuyen Tran

08/23/2022, 5:00 PM
is there a reason why you don’t want to use prefect cloud login?
i

Ilya Galperin

08/23/2022, 5:02 PM
Based on Anna’s response here, it seems like we should be avoiding prefect cloud login for CI/CD? https://prefect-community.slack.com/archives/CL09KU1K7/p1661255009895719?thread_ts=1661254334.222679&amp;cid=CL09KU1K7
k

Khuyen Tran

08/23/2022, 5:18 PM
Do you get a prom for a profile name when using prefect cloud login?
i

Ilya Galperin

08/23/2022, 5:22 PM
We are running this in github actions so there’s no way to respond to a prompt so I’m not positive, but we are also running the command 
prefect profile use default
before 
prefect cloud login
to avoid the prompt.
I think otherwise the prompt will come up yes
k

Khuyen Tran

08/23/2022, 6:22 PM
I think either ways is fine (using prefect cloud login or prefect config set). When using prefect config set, you use the command like the following right? 
prefect profile use xyz
prefect config set PREFECT_API_KEY="xxx"
prefect config set PREFECT_API_KEY="yyy"
i

Ilya Galperin

08/23/2022, 6:29 PM
Exactly
Do we know what might be causing this error?
k

Khuyen Tran

08/23/2022, 6:32 PM
Could you test if you get the same error when running this in your local machine?
i

Ilya Galperin

08/23/2022, 8:03 PM
How do I de-authenticate with cloud CLI? Finding it difficult to test on my local since it’s been authenticated with cloud already and doesn’t seem to want to disconnect.
k

Khuyen Tran

08/23/2022, 8:14 PM
you can just delete the current profile 
prefect profile delete <name-of-your-profile>
Then run 
prefect cloud login -k API-KEY
✅ 1
i

Ilya Galperin

08/23/2022, 8:25 PM
Thanks Khuyen. I’ve tried now on my local as well and just creating a new profile, using 
prefect config set PREFECT_API_KEY/PREFECT_API_URL
and running the below script is generating the same error as described before. 
from prefect.filesystems import S3

def create_storage_block(bucket, flow):
    block_name = f"{flow}-code"

    aws_access_key_id = "xyz123"
    aws_secret_access_key = "xyz123"

    block = S3(
        bucket_path=f"{bucket}/flows/{flow}",
        aws_access_key_id=aws_access_key_id,
        aws_secret_access_key=aws_secret_access_key,
    )
    block.save(block_name, overwrite=True)

    return block_name


if __name__ == "__main__":
    bucket = "mybucket"
    flow = "myflow"

    storage_block = create_storage_block(bucket, flow)
And again if I authenticate with cloud using 
prefect cloud login
, the script works.
k

Khuyen Tran

08/23/2022, 8:30 PM
Can you show the output when running: 
prefect config view
Just to double check
i

Ilya Galperin

08/23/2022, 8:44 PM
PREFECT_PROFILE='xyz2'
PREFECT_API_KEY='my_api_key' (from profile)
PREFECT_API_URL='<https://app.prefect.cloud/accounts/xyz/workspaces/xyz>' (from profile)
I do notice when I run 
prefect cloud login -k my_key
afterwards it’s switching profile back to default from xyz2
prefect cloud login -k my_key                                                                                                          01:43:54 PM
Logged in to Prefect Cloud using profile 'default'.
Workspace is currently set to 'account/workspace'. The workspace can be changed using `prefect cloud workspace set`.
But in absence of doing prefect cloud login I am receiving the JSONDecodeError
t

Tim-Oliver

08/24/2022, 1:44 PM
I added a 
LocalFileSystem
block in the cloud but get the same `JSONDecodeError`when I try to use the block locally. Did you find a solution for this already?
i

Ilya Galperin

08/24/2022, 3:28 PM
Hi Tim — the only “workaround” I’ve found is to actually login to cloud CLI before trying to save our storage block.
3 Views
#prefect-communityJoin Slack
Powered by