Danilo Drobac
06/08/2022, 1:32 PMprefect cloud login --key <API Key I Just Created>
but I keep getting an error that says:
Unable to authenticate. Please ensure your credentials are correct.
Am I missing something?Anna Geller
prefect diagnostics
output? make sure you do:
prefect backend cloud
Danilo Drobac
06/08/2022, 1:36 PMprefect backend cloud
Error: No such command 'backend'.
prefect version
Version: 2.0b6
API version: 0.5.0
Python version: 3.8.10
Git commit: 502ee008
Built: Tue, Jun 7, 2022 4:58 PM
OS/Arch: linux/x86_64
Profile: default
Server type: ephemeral
Server:
Database: sqlite
SQLite version: 3.31.1
Anna Geller
Danilo Drobac
06/08/2022, 1:38 PMAnna Geller
Danilo Drobac
06/08/2022, 1:38 PMAnna Geller
Danilo Drobac
06/08/2022, 1:39 PMAnna Geller
prefect work-queue create default
prefect agent start default
to create a default work queue and start a default agent - this way any deployment you create will create flow runs that will be picked up by this agent:
prefect deployment create yourflow.py
Danilo Drobac
06/08/2022, 1:45 PMstorage
to that location, we would somehow have access to the flow code. Is that correct?
If so, should I be able to deploy directly from the VM, or would it still require me cloning the repo and essentially running the code locally?Anna Geller
name: CI
on:
push:
branches: [ main ]
jobs:
list-flows:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v3
- id: set-matrix
run: echo "::set-output name=matrix::$(ls flows/*.py | jq -R -s -c 'split("\n")[:-1]')"
deploy:
needs: list-flows
runs-on: ubuntu-latest
env:
API_KEY: ${{ secrets.PREFECT_API_KEY}}
strategy:
matrix:
flows: ${{ fromJson(needs.list-flows.outputs.matrix) }}
# flows: [ hello_flow.py, hello_flow_2.py ]
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.9
uses: actions/setup-python@v3
with:
python-version: 3.9
- name: Dev dependencies
run: |
pip install gcsfs
pip install -U "prefect>=2.0b"
- name: Login to Prefect Cloud 2.0
run: prefect cloud login --key $API_KEY --workspace 'annaprefect/prod'
- name: Validate Prefect version
run: prefect version
# Copy and paste the contents of the Service Account JSON file into the secret <https://github.com/marketplace/actions/authenticate-to-google-cloud#authenticating-via-service-account-key-json-1>
- name: Authenticate to GCP
uses: 'google-github-actions/auth@v0.7.1'
with:
credentials_json: '${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
create_credentials_file: true
- name: Create deployment
run: prefect deployment create ${{ matrix.flows }}
and example flow in a flows/ directory
import platform
from prefect import task, flow
from prefect import get_run_logger
from prefect.deployments import DeploymentSpec
from prefect.blocks.storage import FileStorageBlock
@task
def say_hi():
logger = get_run_logger()
<http://logger.info|logger.info>("Hello world!")
@task
def print_platform_info():
logger = get_run_logger()
<http://logger.info|logger.info>(
"Platform information: IP = %s, Python = %s, Platform type = %s, System Version = %s",
platform.node(),
platform.python_version(),
platform.platform(),
platform.version(),
)
@flow
def hello_flow_2():
hi = say_hi()
print_platform_info(wait_for=[hi])
DeploymentSpec(
name="gcs",
flow=hello_flow_2,
flow_storage=FileStorageBlock(base_path="<gcs://prefect-orion/flows>"),
)
if __name__ == "__main__":
hello_flow_2()
Danilo Drobac
06/08/2022, 2:08 PMflows
folder.
If you changed your structure to have a separate deployments
folder you would simply change the path in the action step?
And with that flow script, you're directly specifying GCS as the location for storage, if it wasn't done there, we'd need an additional step in the GitHub Actions script to add the storage details? Is that correct/possible?Anna Geller
Danilo Drobac
06/09/2022, 8:26 PMuses: {builder}
) which is just a Dockerfile that does all the pre-installations of gcsfs and prefect>=2.0b, and I've validated that running prefect version
with that builder returns what I would expect.
When I run the prefect cloud login --key {key} --workspace {workspace}
command, I get a timeout error that happens within ~10seconds of the step running. It can't be a timeout error because of the BuildStep so must be something else.
I've validated that the same command works correctly and as expected when run locally.
Have you ever experienced this or do you have any ideas?
Starting Step #2 - "Login to Prefect Cloud 2.0"
Step #2 - "Login to Prefect Cloud 2.0": Already have image (with digest): <http://gcr.io/nzyte-modern-data-stack/prefect-builder|gcr.io/nzyte-modern-data-stack/prefect-builder>
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 186, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0": addr_obj = ip_address(remote_host)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/ipaddress.py", line 53, in ip_address
Step #2 - "Login to Prefect Cloud 2.0": raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
Step #2 - "Login to Prefect Cloud 2.0": ValueError: '<http://api-beta.prefect.io|api-beta.prefect.io>' does not appear to be an IPv4 or IPv6 address
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0": stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0": gai_res = await getaddrinfo(
Step #2 - "Login to Prefect Cloud 2.0": asyncio.exceptions.CancelledError
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0": yield
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0": stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
Step #2 - "Login to Prefect Cloud 2.0": raise TimeoutError
Step #2 - "Login to Prefect Cloud 2.0": TimeoutError
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0": yield
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0": resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0": raise exc
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0": response = await connection.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0": raise exc
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0": stream = await self._connect(request)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
Step #2 - "Login to Prefect Cloud 2.0": stream = await self._network_backend.connect_tcp(**kwargs)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0": return await self._backend.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0": stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0": self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0": raise to_exc(exc)
Step #2 - "Login to Prefect Cloud 2.0": httpcore.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": The above exception was the direct cause of the following exception:
Step #2 - "Login to Prefect Cloud 2.0":
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
Step #2 - "Login to Prefect Cloud 2.0": return fn(*args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 122, in wrapper
Step #2 - "Login to Prefect Cloud 2.0": return run_async_in_new_loop(async_fn, *args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 69, in run_async_in_new_loop
Step #2 - "Login to Prefect Cloud 2.0": return anyio.run(partial(__fn, *args, **kwargs))
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
Step #2 - "Login to Prefect Cloud 2.0": return asynclib.run(func, *args, **backend_options)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
Step #2 - "Login to Prefect Cloud 2.0": return native_run(wrapper(), debug=debug)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
Step #2 - "Login to Prefect Cloud 2.0": return loop.run_until_complete(main)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Step #2 - "Login to Prefect Cloud 2.0": return future.result()
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
Step #2 - "Login to Prefect Cloud 2.0": return await func(*args)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 204, in login
Step #2 - "Login to Prefect Cloud 2.0": workspaces = await client.read_workspaces()
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 82, in read_workspaces
Step #2 - "Login to Prefect Cloud 2.0": return await self.get("/me/workspaces")
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 102, in get
Step #2 - "Login to Prefect Cloud 2.0": res = await self._client.get(route, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
Step #2 - "Login to Prefect Cloud 2.0": return await self.request(
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
Step #2 - "Login to Prefect Cloud 2.0": return await self.send(request, auth=auth, follow_redirects=follow_redirects)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
Step #2 - "Login to Prefect Cloud 2.0": response = await self._send_handling_auth(
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
Step #2 - "Login to Prefect Cloud 2.0": response = await self._send_handling_redirects(
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
Step #2 - "Login to Prefect Cloud 2.0": response = await self._send_single_request(request)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
Step #2 - "Login to Prefect Cloud 2.0": response = await transport.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0": resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0": self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0": File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0": raise mapped_exc(message) from exc
Step #2 - "Login to Prefect Cloud 2.0": httpx.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0": An exception occurred.
Anna Geller
Danilo Drobac
06/10/2022, 11:55 AMAnna Geller