https://prefect.io logo
Powered by
Title
d

Danilo Drobac

06/08/2022, 1:32 PM
I'm trying to run 
prefect cloud login --key <API Key I Just Created>
but I keep getting an error that says: 
Unable to authenticate. Please ensure your credentials are correct.
Am I missing something?
✅ 1
a

Anna Geller

06/08/2022, 1:35 PM
can you share your 
prefect diagnostics
output? make sure you do: 
prefect backend cloud
also, if you're just getting started, you may start directly with Cloud 2.0
d

Danilo Drobac

06/08/2022, 1:36 PM
prefect backend cloud
Error: No such command 'backend'.
prefect version
Version:             2.0b6
API version:         0.5.0
Python version:      3.8.10
Git commit:          502ee008
Built:               Tue, Jun 7, 2022 4:58 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.31.1
a

Anna Geller

06/08/2022, 1:37 PM
ok, so it looks like you are on Prefect 2.0 not on 1.0 - which is great!
let me try to replicate it to make sure it's not an issue with Cloud 2.0
d

Danilo Drobac

06/08/2022, 1:38 PM
I see. Looks like I've created an account for Cloud 1.0 rather than 2.0
a

Anna Geller

06/08/2022, 1:38 PM
oh that explains a lot 😄
d

Danilo Drobac

06/08/2022, 1:38 PM
Certainly does 🤦
a

Anna Geller

06/08/2022, 1:38 PM
for 2.0b6 - you need beta.prefect.io
for Prefect 1.0 you use cloud.prefect.io
it's totally justifiable that it may be confusing - that's the main reason I created the above Discourse page
d

Danilo Drobac

06/08/2022, 1:39 PM
Okay, I'm logged in now. That makes sense. Let me go back through that Discourse page and see if I can make more sense of it now I've put some stuff in action
🙌 1
One additional question - I'm trying to understand how the agents / queues work on a VM. I have Cloud 2.0 as the UI component. I've set up a VM and installed Prefect 2.0 on there and set the storage to a GCS bucket that I've got. Am I right in thinking that I now have to create a deployment (locally) to Cloud 2.0 and then I can use the VM as an agent?
a

Anna Geller

06/08/2022, 1:42 PM
exactly!
✅ 1
you can do: 
prefect work-queue create default
prefect agent start default
to create a default work queue and start a default agent - this way any deployment you create will create flow runs that will be picked up by this agent: 
prefect deployment create yourflow.py
d

Danilo Drobac

06/08/2022, 1:45 PM
Another one for you - my storage bucket contains all of the flow code that I want to keep maintained (it's set up to mirror my GitHub repo with the code in). My assumption was that by setting the 
storage
to that location, we would somehow have access to the flow code. Is that correct? If so, should I be able to deploy directly from the VM, or would it still require me cloning the repo and essentially running the code locally?
And to give you the reason why I'm trying to understand all of these components, it's ultimately so that I can work out what my CI/CD needs to look like. For example, when my code changes for a flow and I push the change to GitHub, I want to make that: 1. If it's an existing deployment, I want to make sure it is updated to the new code 2. If it's a new deployment, I want to have the creation of it automated Is this a normal workflow of how the product works?
a

Anna Geller

06/08/2022, 1:58 PM
are you using S3?
wait you said GCS, sorry
🙊 1
here is example of a working GitHub Actions 
name: CI
on:
  push:
    branches: [ main ]
jobs:
  list-flows:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v3
      - id: set-matrix
        run: echo "::set-output name=matrix::$(ls flows/*.py | jq -R -s -c 'split("\n")[:-1]')"
  deploy:
    needs: list-flows
    runs-on: ubuntu-latest
    env:
      API_KEY: ${{ secrets.PREFECT_API_KEY}}
    strategy:
      matrix:
        flows: ${{ fromJson(needs.list-flows.outputs.matrix) }}
        # flows: [ hello_flow.py, hello_flow_2.py ]
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.9
        uses: actions/setup-python@v3
        with:
          python-version: 3.9
      - name: Dev dependencies
        run: |
          pip install gcsfs
          pip install -U "prefect>=2.0b"
      - name: Login to Prefect Cloud 2.0
        run: prefect cloud login --key $API_KEY --workspace 'annaprefect/prod'
      - name: Validate Prefect version
        run: prefect version
      # Copy and paste the contents of the Service Account JSON file into the secret <https://github.com/marketplace/actions/authenticate-to-google-cloud#authenticating-via-service-account-key-json-1>
      - name: Authenticate to GCP
        uses: 'google-github-actions/auth@v0.7.1'
        with:
          credentials_json: '${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
          create_credentials_file: true
      - name: Create deployment
        run: prefect deployment create ${{ matrix.flows }}
and example flow in a flows/ directory 
import platform
from prefect import task, flow
from prefect import get_run_logger
from prefect.deployments import DeploymentSpec
from prefect.blocks.storage import FileStorageBlock


@task
def say_hi():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello world!")


@task
def print_platform_info():
    logger = get_run_logger()
    <http://logger.info|logger.info>(
        "Platform information: IP = %s, Python = %s, Platform type = %s, System Version = %s",
        platform.node(),
        platform.python_version(),
        platform.platform(),
        platform.version(),
    )


@flow
def hello_flow_2():
    hi = say_hi()
    print_platform_info(wait_for=[hi])


DeploymentSpec(
    name="gcs",
    flow=hello_flow_2,
    flow_storage=FileStorageBlock(base_path="<gcs://prefect-orion/flows>"),
)

if __name__ == "__main__":
    hello_flow_2()
d

Danilo Drobac

06/08/2022, 2:08 PM
Well that GitHub actions script is genius. Now to see whether it can be reproduced in Google Cloud Build 😆 So that literally re-creates the deployments for everything that's found as a Python script in the 
flows
folder. If you changed your structure to have a separate 
deployments
folder you would simply change the path in the action step? And with that flow script, you're directly specifying GCS as the location for storage, if it wasn't done there, we'd need an additional step in the GitHub Actions script to add the storage details? Is that correct/possible?
a

Anna Geller

06/08/2022, 2:13 PM
#1 yes exactly - it looks for deployment specs, not for flows #2 if you don't explicitly specify storage, Prefect will use your default storage and will upload your flow there during deployment creation - but note it will be uploaded to cloud storage as a plain text file with uuid, rather than as python file with custom name - more on storage here https://discourse.prefect.io/t/when-should-i-use-filestorageblock-vs-specific-cloud-pla[…]block-googlecloudstorageblock-or-azureblobstorageblock/924
✅ 1
d

Danilo Drobac

06/09/2022, 8:26 PM
Hey. I've been working on converting the GitHub actions script to work with Cloud Build and I'm making good progress, but I'm getting stuck when I get to the authentication with Prefect Cloud. I've created a builder (the equivalent of the GitHub version of 
uses: {builder}
) which is just a Dockerfile that does all the pre-installations of gcsfs and prefect>=2.0b, and I've validated that running 
prefect version
with that builder returns what I would expect. When I run the 
prefect cloud login --key {key} --workspace {workspace}
command, I get a timeout error that happens within ~10seconds of the step running. It can't be a timeout error because of the BuildStep so must be something else. I've validated that the same command works correctly and as expected when run locally. Have you ever experienced this or do you have any ideas? 
Starting Step #2 - "Login to Prefect Cloud 2.0"
Step #2 - "Login to Prefect Cloud 2.0": Already have image (with digest): <http://gcr.io/nzyte-modern-data-stack/prefect-builder|gcr.io/nzyte-modern-data-stack/prefect-builder>
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 186, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     addr_obj = ip_address(remote_host)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/ipaddress.py", line 53, in ip_address
Step #2 - "Login to Prefect Cloud 2.0":     raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
Step #2 - "Login to Prefect Cloud 2.0": ValueError: '<http://api-beta.prefect.io|api-beta.prefect.io>' does not appear to be an IPv4 or IPv6 address
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     gai_res = await getaddrinfo(
Step #2 - "Login to Prefect Cloud 2.0": asyncio.exceptions.CancelledError
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     yield
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     raise TimeoutError
Step #2 - "Login to Prefect Cloud 2.0": TimeoutError
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     yield
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     raise exc
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     response = await connection.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     raise exc
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     stream = await self._connect(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
Step #2 - "Login to Prefect Cloud 2.0":     stream = await self._network_backend.connect_tcp(**kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     return await self._backend.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     raise to_exc(exc)
Step #2 - "Login to Prefect Cloud 2.0": httpcore.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": The above exception was the direct cause of the following exception:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return fn(*args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 122, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return run_async_in_new_loop(async_fn, *args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 69, in run_async_in_new_loop
Step #2 - "Login to Prefect Cloud 2.0":     return anyio.run(partial(__fn, *args, **kwargs))
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
Step #2 - "Login to Prefect Cloud 2.0":     return asynclib.run(func, *args, **backend_options)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
Step #2 - "Login to Prefect Cloud 2.0":     return native_run(wrapper(), debug=debug)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
Step #2 - "Login to Prefect Cloud 2.0":     return loop.run_until_complete(main)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Step #2 - "Login to Prefect Cloud 2.0":     return future.result()
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return await func(*args)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 204, in login
Step #2 - "Login to Prefect Cloud 2.0":     workspaces = await client.read_workspaces()
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 82, in read_workspaces
Step #2 - "Login to Prefect Cloud 2.0":     return await self.get("/me/workspaces")
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 102, in get
Step #2 - "Login to Prefect Cloud 2.0":     res = await self._client.get(route, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
Step #2 - "Login to Prefect Cloud 2.0":     return await self.request(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
Step #2 - "Login to Prefect Cloud 2.0":     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_handling_auth(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_handling_redirects(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_single_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
Step #2 - "Login to Prefect Cloud 2.0":     response = await transport.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     raise mapped_exc(message) from exc
Step #2 - "Login to Prefect Cloud 2.0": httpx.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0": An exception occurred.
a

Anna Geller

06/10/2022, 11:53 AM
ok, given that you asked about the same in a separate thread, I'll treat this one as closed for the future, no need to open multiple threads for the same issue - we had to talk to figure out what you are doing in this new thread while it was clear from here that you are running CI from Google Cloud Build - at least can you share a link to the previous conversation next time? this would be helpful
d

Danilo Drobac

06/10/2022, 11:55 AM
Sorry, assumed you'd missed it and wanted to have the question as the initial message..
a

Anna Geller

06/10/2022, 12:03 PM
all good, it's just a lot of threads and we need to give everyone the chance 😄 will respond soon
7 Views
#prefect-communityJoin Slack
Powered by