https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • i

    Ian Andres Etnyre Mercader

    09/08/2022, 7:59 PM
    Hi guys, I'am using
    prefect 2.3.1
    ,
    prefect-dask: 0.2.0
    and got several errors running code similar to this pseudo code :
    @task
    def dask_context(item):
        name="myblock-name"
        config = await JSON.load(name)
    
    @flow(task_runner=DaskTaskRunner())
    def do_stuff():
       with tags("only_4"):
           result_list = dask_context.map(data_list)
    I got this Exception multiple times after it running for a while (sometimes minutes, sometimes hours):
    Encountered exception during execution:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read
        return await self._stream.receive(max_bytes=max_bytes)
      File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
        await self._protocol.read_event.wait()
      File "/usr/local/lib/python3.9/asyncio/locks.py", line 226, in wait
        await fut
    asyncio.exceptions.CancelledError
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
        yield
      File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 35, in read
        return b""
      File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
        raise TimeoutError
    TimeoutError
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
        yield
      File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
        resp = await self._pool.handle_async_request(req)
      File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
        raise exc
      File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
        response = await connection.handle_async_request(request)
      File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
        return await self._connection.handle_async_request(request)
      File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 105, in handle_async_request
        raise exc
      File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 84, in handle_async_request
        ) = await self._receive_response_headers(**kwargs)
      File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 148, in _receive_response_headers
        event = await self._receive_event(timeout=timeout)
      File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 177, in _receive_event
        data = await self._network_stream.read(
      File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 35, in read
        return b""
      File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
        raise to_exc(exc)
    httpcore.ReadTimeout
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1185, in orchestrate_task_run
        result = await task.fn(*args, **kwargs)
      File "/tmp/tmpq25d58myprefect/lib/dask_context.py", line 14, in _dask_task
        with await get_context(flow_run.parameters[block_name]):
      File "/tmp/tmpq25d58myprefect/./lib/config_context.py", line 40, in get_context
        config = await JSON.load(name)
      File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 606, in load
        block_document = await client.read_block_document_by_name(
      File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 1268, in read_block_document_by_name
        response = await self._client.get(
      File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
        return await self.request(
      File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
        return await self.send(request, auth=auth, follow_redirects=follow_redirects)
      File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 257, in send
        await super().send(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
        response = await self._send_handling_auth(
      File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
        response = await self._send_handling_redirects(
      File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
        response = await self._send_single_request(request)
      File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
        response = await transport.handle_async_request(request)
      File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
        resp = await self._pool.handle_async_request(req)
      File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
        raise mapped_exc(message) from exc
    httpx.ReadTimeout
    I started running this code with a tag concurrency limit of 20 then switched to 4. The more threads the exception was raised faster. Currently running on a server of 44 cores and 256 GB.
    ✅ 1
    f
    j
    • 3
    • 5
  • b

    Bradley Hurley

    09/08/2022, 8:35 PM
    Hi - Prefect Experts, Im running into an error that indicates I am unable to schedule a flow that has required parameters. Is that expected? 🧵
    1️⃣ 1
    ✅ 1
    k
    • 2
    • 13
  • b

    Brad Clark

    09/08/2022, 9:33 PM
    Is there something that has to be done to my account to enable login through the CLI? I have created an API key but when I run
    prefect cloud login
    i get
    Unable to authenticate with Prefect Cloud. Please ensure your credentials are correct.
    ✅ 1
    • 1
    • 1
  • f

    Felix Tremblay

    09/08/2022, 9:33 PM
    Hi all, I was having issues with the logging in Prefect. I wanted to share my solution in case it helps someone else. Issue 1 - Prefect logger does not work outside of Prefect context Issue 2 - Prefect logger needs to have get_run_logger() in every functions Solution (proof-of-concept): See https://discourse.prefect.io/t/hybrid-logger-a-logger-that-works-both-inside-and-outsite-of-a-prefect-context/1559 @Taylor Curran
    :wood: 3
    :blob-attention-gif: 4
    ✅ 2
    :marvin: 4
    😛anda-dancing: 4
    t
    a
    • 3
    • 3
  • a

    Aaron Goebel

    09/08/2022, 10:57 PM
    Is it possible to get a replica of the database of historical runs etc that Prefect cloud manages? Or something like incremental backups provided to end users
    ✅ 1
    a
    • 2
    • 1
  • y

    Yusuf

    09/09/2022, 1:06 AM
    How do I add mapped tasks in the imperative API style? so for example:
    flow = Flow("My imperative flow!")
    
    # define some new tasks
    name = Parameter("name")
    second_add = add.copy()
    
    # add our tasks to the flow
    flow.add_task(add)
    flow.add_task(second_add)
    flow.add_task(say_hello)
    
    # create non-data dependencies so that `say_hello` waits for `second_add` to finish.
    say_hello.set_upstream(second_add, flow=flow)
    
    # create data bindings
    add.bind(x=1, y=2, flow=flow)
    second_add.bind(x=add, y=100, flow=flow)
    say_hello.bind(person=name, flow=flow)
    If I wanted to give the
    add
    task a mapped input?
    👀 1
    j
    • 2
    • 5
  • y

    Yousef Hosny

    09/09/2022, 1:26 AM
    Hi all, I am getting this error when building a deployment..
    ✅ 1
    a
    • 2
    • 2
  • y

    Yaron Levi

    09/09/2022, 4:08 AM
    Hi 👋 I am trying to use the GitHub repo feature that was added in 2.3.0 I have this code for creating the deployment (🔥*which doesn’t work*🔥):
    from weekly_bonus_calc import weekly_bonus_calc
    from prefect.deployments import Deployment
    from prefect.filesystems import GitHub
    
    storage = GitHub(
        repository="<https://github.com/yuvital/yuvital-prefect-github.git>"
    )
    
    deployment = Deployment.build_from_flow(
        flow=weekly_bonus_calc,
        name="weekly-bonus-calc",    
        work_queue_name="work-queue",
        storage=storage,
        schedule={'cron': '*/5 * * * *', 'timezone': "Asia/Jerusalem"}
    )
    
    deployment.apply()
    Also, I’ve created a GitHub block via the UI (attaching a screenshot). Can you point out the correct way to setup this. Another thing I don’t understand is where are the Git credentials should go since this is obviously a private repo. Thank you for any help on the issue 🙏
    ✅ 1
    j
    • 2
    • 1
  • a

    ash

    09/09/2022, 6:17 AM
    hello everyone 👋, I am using prefect server 1.0 deployed on k8 with git as storage. I want to try event based triggering of prefect flows by a third party api , Is it possible in prefect server, if yes then how ?
    ✅ 1
    n
    • 2
    • 3
  • b

    Bal Raj

    09/09/2022, 6:27 AM
    hello everyone, I am using create_flow_run() task in a parent flow to trigger a flow.So when this flow failed in halfway.So my parent flow should restart it instead creating a new flow all together. but how to achieve the former case?
    ✅ 1
    s
    • 2
    • 1
  • m

    Malavika S Menon

    09/09/2022, 6:29 AM
    def random_fn():
       print("Random")
    
    @flow
    def main_flow():
            prefect_flow = Flow(name="random_flow", fn=random_fn)
            deployment = Deployment.build_from_flow(
                flow=prefect_flow,
                name=random_deploy,
                version=1,
                work_queue_name="test-queue",
            )
            deployment.apply()
    I'm trying to build a deployment for a flow that is created dynamically. While the deployment 'random_flow' gets created successfully, when I try to run the same, I get this error.
    Flow could not be retrieved from deployment.
    Traceback (most recent call last):
      File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/engine.py", line 256, in retrieve_flow_then_begin_flow_run
        flow = await load_flow_from_flow_run(flow_run, client=client)
      File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/client.py", line 103, in with_injected_client
        return await fn(*args, **kwargs)
      File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/deployments.py", line 54, in load_flow_from_flow_run
        await storage_block.get_directory(from_path=deployment.path, local_path=".")
      File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/filesystems.py", line 134, in get_directory
        shutil.copytree(from_path, local_path, dirs_exist_ok=True)
      File "/usr/lib/python3.8/shutil.py", line 555, in copytree
        with os.scandir(src) as itr:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpbkxm8cdtprefect'
    What can be done to link this flow to the deployment properly and get it to run through the Orion UI?
    s
    • 2
    • 1
  • m

    Matthias

    09/09/2022, 6:55 AM
    Anyone in the community that is using Prefect for CI/CD? And if so, how do you like it so far?
    r
    • 2
    • 1
  • m

    Michael Law

    09/09/2022, 8:05 AM
    Hey guys, I might be missing something in v2, but in v1 I belive we can re-run just the failed tasks from a flow run, can we do the same in v2?
    👀 1
    ✅ 1
    r
    • 2
    • 7
  • b

    Bal Raj

    09/09/2022, 10:55 AM
    What are other ways to restart a flow apart from restarting in ui?
    ✅ 1
    a
    • 2
    • 5
  • g

    gertjan

    09/09/2022, 11:23 AM
    How can you stream logs to the UI from functions called in a Task that are using their own logger outside prefect? I can see those logs in the terminal, but never in the UI. I only can see logs that are defined with the
    get_run_logger
    👀 1
    j
    • 2
    • 3
  • v

    Viet Nguyen

    09/09/2022, 11:24 AM
    Hi @Anna Geller and everyone, I wonder if there is API endpoint to check if the Agent is busy executing flows or in Idle mode? Thank you.
    ✅ 1
    a
    • 2
    • 10
  • s

    Slackbot

    09/09/2022, 12:04 PM
    This message was deleted.
  • v

    Vadym Dytyniak

    09/09/2022, 12:33 PM
    Hi. Should I install s3fs in addition to prefect to run the flow, because I see an error
    ModuleNotFoundError: No module named 's3fs'
    ?
    ✅ 1
    a
    • 2
    • 3
  • v

    Vadym Dytyniak

    09/09/2022, 12:44 PM
    I receive an error that agent can't create a queue even if it exists, I use service account API key with role member. Should it be role with more permissions?
    m
    • 2
    • 11
  • d

    Dominik Wagner

    09/09/2022, 1:10 PM
    Prefect 2.3.1 - CI/CD: I run a freshly deployed flow at the end of my CI/CD pipeline, and would like to know whether or not the run was successful. My current idea is to poll the API for the state of the submitted run until it reaches a terminal state. Has anyone implemented a solution like this, or do you have another idea on how to solve this? 🙏
    ✅ 1
    s
    r
    • 3
    • 4
  • v

    Vadym Dytyniak

    09/09/2022, 1:13 PM
    Hi. Does Prefect 2 support extras like previous version?(prefect[aws,kubernetes])
    ✅ 1
    j
    • 2
    • 2
  • m

    Mark Li

    09/09/2022, 1:56 PM
    Are there any details anyone needs me to fill in? Is this the right place to be posting this? Don’t know how we like to “bump” messages here…
    n
    • 2
    • 9
  • b

    Benjamin Cerigo

    09/09/2022, 2:25 PM
    Hi @Chris Reuter and everyone. Is there someone that can help with answering this issues that I posted: https://github.com/anna-geller/prefect-docker-deployment/issues/1?
    j
    k
    • 3
    • 3
  • k

    Kei Fung Dennis Tai

    09/09/2022, 2:36 PM
    Hi prefect experts, I'm two days new since discovering prefect, I have a question about deployments. So If I understand correctly, unless I register a flow as a deployment, I can't trigger the flow in the orion UI Is that right? I saw docs that taught how to register a deployment (CLI or Python), but it looks like either method I can run multiple times and it will register the flow as deployments multiple times. Is there a way to create a file that simply "define" the list of deployments like flows does right now? Also how do I remove a deployment? is it through the orion UI only? Also, there seems to be no way to trigger a part of the flow (e.g. if the flow runs subflows 1,2,3,4,5 sequentially, there's no way to just run 3,4,5). Is that correct? Thanks!
    ✅ 1
    s
    j
    • 3
    • 5
  • r

    Robin Weiß

    09/09/2022, 3:20 PM
    Hey dear Prefectionists, I am running into weird errors with my K8s cluster workloads. The flows actually keep crashing with a 502 Bad Gateway error that mentions a Prefect Cloud URI:
    prefect.exceptions.PrefectHTTPStatusError: Server error '502 Bad Gateway' for url '<https://api.prefect.cloud/api/accounts/><ACCOUNT>/workspaces/<WORKSPACE>/logs/'
    For more information check: <https://httpstatuses.com/502>
    Worker information:
        Approximate queue length: 0
        Pending log batch length: 1
        Pending log batch size: 360
    The log worker will attempt to send these logs again in 2.0s
    Is it possible that there are too many log messages sent and the Prefect Cloud API can not respond in time? Cheers!
    ✅ 1
    g
    • 2
    • 1
  • a

    Amol Shirke

    09/09/2022, 4:01 PM
    Hello, Can't multiple users use same prefect db? I have setup postgress db and connecting with dbuser and password. User who start server first time it works for him however next user who try to start server connecting to same db there is error alembic.until.exc.CommandErro:Can't locate revision identified by 'd3....'
    👀 1
    ✅ 1
    g
    • 2
    • 1
  • j

    Javier Smith

    09/09/2022, 4:54 PM
    Hi! can a Prefect project be moved between teams? I've created a project in a personal account and recently joined a company account (with other members) but can't find info anywhere about how to make this movement (currently using Prefect 1.0)
    ✅ 1
    j
    • 2
    • 3
  • m

    Mansour Zayer

    09/09/2022, 5:35 PM
    Hello. In Prefect 1, I had a few flows stuck in Submitted for more than 1 hour, but Lazarus never did anything (although it's enabled for these flows). Is there a way to change Lazarus settings, or make sure it's working? (In the UI it's only enable/disable) I couldn't find any info in the documentation. Thank you
    ✅ 1
    r
    • 2
    • 1
  • n

    Nathaniel Russell

    09/09/2022, 5:58 PM
    Do prefect 2's blocks have any sort of locking functionality? (imagine a queue-like block with some flows adding to it and others removing, how could they maintain reliability that no item gets lost?)
    👀 2
    a
    • 2
    • 1
  • i

    Ilya Galperin

    09/09/2022, 6:11 PM
    Hi all - is there a way to enable verbose debugging on an agent in Prefect 2, or another method to troubleshoot an agent seemingly not connecting to a workspace?
    👀 1
    j
    c
    • 3
    • 74
Powered by Linen
Title
i

Ilya Galperin

09/09/2022, 6:11 PM
Hi all - is there a way to enable verbose debugging on an agent in Prefect 2, or another method to troubleshoot an agent seemingly not connecting to a workspace?
👀 1
j

Jean Luciano

09/09/2022, 6:45 PM
Hello @Ilya Galperin, we don't have a way to enable more verbose logs on the agent. When the agent is started, does it say that it's looking for work from the the right work queue? i.e
Agent started! Looking for work from queue(s): workque-chosen...
i

Ilya Galperin

09/09/2022, 6:47 PM
Hi Jean — it is pointing to the correct queue and the correct API url but we can’t tell if it’s actually succeeding in authenticating
j

Jean Luciano

09/09/2022, 7:17 PM
Where is your agent running?
i

Ilya Galperin

09/09/2022, 7:33 PM
It is running on an EKS cluster
c

Christopher Boyd

09/12/2022, 5:13 PM
Hi @Ilya Galperin - this can be verified pretty quickly - if you are passing in a secret for your API key, you can modify the secret to something you know to be incorrect - the agent will then fail to connect successfully with a 401/403
if you aren’t seeing a 401/403, then it should be successfully connected. You can also review the api/agent logs individually
Why do you suspect it’s incorrect?
i

Ilya Galperin

09/12/2022, 5:26 PM
Thanks Christopher. I’m not entirely sure if it’s the API key or something else. We are not seeing a 401/403 error. The agent log looks normal:
prefect-agent-8b76465df-sq4g7 agent Starting v2.3.2 agent connected to 
prefect-agent-8b76465df-sq4g7 agent <https://api.prefect.cloud/api/accounts/ACCOUNTID/work>
prefect-agent-8b76465df-sq4g7 agent spaces/WORKSPACEID...
prefect-agent-8b76465df-sq4g7 agent 
prefect-agent-8b76465df-sq4g7 agent   ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
prefect-agent-8b76465df-sq4g7 agent  | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
prefect-agent-8b76465df-sq4g7 agent  |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
prefect-agent-8b76465df-sq4g7 agent  |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|
prefect-agent-8b76465df-sq4g7 agent 
prefect-agent-8b76465df-sq4g7 agent 
prefect-agent-8b76465df-sq4g7 agent Agent started! Looking for work from queue(s): main...
However, no flows from our “main” queue are being picked up by this agent which makes me think there might be a communication problem between the agent and the server, whether it’s an incorrect API key or something else. Is there some other logs or mechanism we can use to validate that it is connecting successfully… you mentioned API logs?
One reason I thought it might be the API key or something with our configuration is that we didn’t see any errors being thrown when pointing the agent to an entirely incorrect URL i.e. google.com so it doesn’t seem like it cares if there’s a real connection?
c

Christopher Boyd

09/12/2022, 6:07 PM
How did you deploy your agent ? Is this in kubernetes , or a vm or local ?
It should definitely care if it's not a valid url or it's an invalid api key , but I can't tell much more without seeing the configuration unfortunately
i

Ilya Galperin

09/12/2022, 6:09 PM
It is in Kubernetes. We’re doing some more troubleshooting right now to see if this is a networking configuration issue with the cluster this agent is running on. If that were to be the case, would we still see an error? And if so, what error?
We definitely did not see an error when pointing to google.com
c

Christopher Boyd

09/12/2022, 6:09 PM
How are you viewing / retrieving logs ?
i

Ilya Galperin

09/12/2022, 6:10 PM
Are we sure there is logging for this on 2.3.2? If so, it might be missing some edge case that we’re running into?
We are using kubectl to see the logs of the pod the agent is running on
c

Christopher Boyd

09/12/2022, 6:10 PM
Can you sure your deployment / manifest you used to deploy this agent ?
You can redact api key / workspace
i

Ilya Galperin

09/12/2022, 6:11 PM
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-agent
  namespace: prefect2
spec:
  selector:
    matchLabels:
      app: prefect-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      serviceAccountName: default
      containers:
      - name: agent
        image: prefecthq/prefect:${var.prefect_tag}
        command: ["prefect", "agent", "start", "-q", "main"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: ${var.prefect_api_url}
          - name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: prefect-cloud 
                key: api_key
${var.prefect_tag}
in this case resolves to “2.3.2-python3.10”
and if you plug in prefect_api_url = “http://google.com” in this configuration we don’t get any error on the agent pod
c

Christopher Boyd

09/12/2022, 6:13 PM
Ok give me a few minutes
👍 1
i

Ilya Galperin

09/12/2022, 6:13 PM
Thank you
c

Christopher Boyd

09/12/2022, 6:24 PM
I’m deploying a fresh instance currently wiht the following:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orion
  namespace: prefect2
spec:
  selector:
    matchLabels:
      app: orion
  replicas: 1  # We're using SQLite, so we should only run 1 pod
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:2.3.2-python3.10
        command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 4200
      - name: agent
        image: prefecthq/prefect:2.3.2-python3.10
        command: ["prefect", "agent", "start", "kubernetes"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: <https://api.prefect.cloud/api/accounts/><accountid>/workspaces/<workspaceid>
          - name: PREFECT_API_KEY
            value: bad_value

---
apiVersion: v1
kind: Service
metadata:
  name: orion
  namespace: prefect2
  labels:
    app: orion
spec:
  ports:
    - port: 4200
      protocol: TCP
  selector:
    app: orion
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  namespace: prefect2
  name: flow-runner
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: flow-runner-role-binding
  namespace: prefect2
subjects:
- kind: ServiceAccount
  name: default
  namespace: prefect2
roleRef:
  kind: Role
  name: flow-runner
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
once this comes up, I’ll update with bad api url (e.g. like you said google), and verify with a good api key afterwards for both
I received the following for the agent:
prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url '<https://api.prefect.cloud/api/accounts/><redact>/workspaces/<redact>/work_queues/name/kubernetes'
Response: {'detail': 'Invalid authentication credentials'}
For more information check: <https://httpstatuses.com/401>
An exception occurred.
The following for the api:
Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>

View the API reference documentation at <http://0.0.0.0:4200/docs>

Check out the dashboard at <http://0.0.0.0:4200>



INFO:     Started server process [9]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on <http://0.0.0.0:4200> (Press CTRL+C to quit)
updating to a good API key first, then to a bad api url
with a good api key (same manifest, just modified the run time api key):
Agents now support multiple work queues. Instead of passing a single argument, 
provide work queue names with the `-q` or `--work-queue` flag: `prefect agent 
start -q kubernetes`

Starting v2.3.2 agent connected to 
<https://api.prefect.cloud/api/accounts/redact/work>
spaces/redact...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): kubernetes...
with a bad url:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 88, in get_work_queues
    work_queue = await self.client.create_work_queue(name=name)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 835, in create_work_queue
    response = await <http://self._client.post|self._client.post>("/work_queues/", json=data)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 279, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 225, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://google.com/work_queues/>'
For more information check: <https://httpstatuses.com/404>
agent communications should be outbound only, the agent will sleep, then check in with the api for work, before sleeping again, so an inbound connection or ingress isn’t required
i

Ilya Galperin

09/12/2022, 6:33 PM
Sorry, I’m not 100% following which test was which. Are you receiving an error when passing in a bad URL but a valid-looking key?
c

Christopher Boyd

09/12/2022, 6:34 PM
I receive a successful configuration with good key + good url. I receive a 401 with a bad key and good url I receive a 404 with a bad url and a good key
✅ 1
i

Ilya Galperin

09/12/2022, 6:34 PM
Also, would the
Service
manifest here have any impact on connectivity between Cloud and the Agent? My understanding is that this would only be for communication between pods spawned by the agent and the agent itself, right?
That is really strange, our bad URL log looked like this…
Starting v2.3.0 agent connected to <http://google.com>...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): main...
(this was on 2.3.0 not sure if that makes a difference)
c

Christopher Boyd

09/12/2022, 6:39 PM
the service manifest should have no relevant, that’s just how the service is being exposed to reach from the outside in
i

Ilya Galperin

09/12/2022, 6:39 PM
Gotcha ok
c

Christopher Boyd

09/12/2022, 6:40 PM
and yes, I receive the same top of stack message with google:
Agents now support multiple work queues. Instead of passing a single argument, 
provide work queue names with the `-q` or `--work-queue` flag: `prefect agent 
start -q kubernetes`

Starting v2.3.2 agent connected to <https://google.com>...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): kubernetes...
the bottom of stack is
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://google.com/work_queues/>'
For more information check: <https://httpstatuses.com/404>
I would say if your api key is correct, and your url is correct (you can verify by checking prefect config view if your workspace is set locally on your cli) then your agent is successfully configured
assuming there are no more detailed logs with errors like above
i

Ilya Galperin

09/12/2022, 6:43 PM
Ya, we are definitely not seeing that in our log 😕
We are thinking it might be a DNS issue on the cluster
I wonder if there is an except somewhere in the agent that is preventing that output
Will update as we continue troubleshooting
c

Christopher Boyd

09/12/2022, 6:44 PM
what you can probably try out of curiosity
is shell exec into one of the pods
and curl the api / auth login
i

Ilya Galperin

09/12/2022, 6:45 PM
ah thats not a bad idea
c

Christopher Boyd

09/12/2022, 6:45 PM
see if you can reach it manually from a pod shell
i

Ilya Galperin

09/12/2022, 6:45 PM
We will try this as well, thank you Christopher
✅ 1
Will report back
Sorry - do we know that the linux distribution included in the agent has curl?
c

Christopher Boyd

09/12/2022, 6:51 PM
I can check, but if not, it can be added pretty quickly to a new image
Ill check
curl isnt native in the image, but prefect is
you shell in, and do prefect cloud login with your api key
i

Ilya Galperin

09/12/2022, 6:54 PM
ah ok
thank you Christopher
c

Christopher Boyd

09/12/2022, 6:55 PM
via -
prefect cloud login --key  <key>
✅ 1
i

Ilya Galperin

09/12/2022, 7:18 PM
Just an update Chris - we tried installing curl on this pod just to check but we’re running into a DNS error which is preventing us from doing so.
# apt-get install curl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package curl
# apt-get update
Err:1 <http://deb.debian.org/debian> bullseye InRelease
  Temporary failure resolving '<http://deb.debian.org|deb.debian.org>'
Which makes us think it’s possible that something in the agent is “eating” a dns-related error which would be surfaced since we’re definitely not connecting to cloud or anything outside the container
c

Christopher Boyd

09/12/2022, 7:22 PM
you can try creating a standalone image outside (no curl , but can use pip to install httpie which is just fancier curl)-
cat DOCKERFILE
FROM prefecthq/prefect:2.32.0-python3.10
RUN pip install httpie
then:
export IMAGE_REGISTRY="my_image_registry"
export PROJECT_NAME="prefect-httpie-3.10"
export PROJECT_VERSION="latest"

echo "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
docker build --platform=linux/amd64 -t "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION" -f ./DOCKERFILE .
docker push "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
✅ 1
then update your image tags with my_image_registry/prefect-httpie-3.10:latest
👍 1
i

Ilya Galperin

09/12/2022, 7:23 PM
I think we’ll probably run into another dns error here either way, but we will check, thanks!
c

Christopher Boyd

09/12/2022, 7:24 PM
no problem - this would all be local on your system
unless the issue is pulling the image
http usage is functionally the same as curl, just better presentation:
http <https://api.prefect.io>
HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Length: 18
Content-Type: text/html; charset=utf-8
Date: Mon, 12 Sep 2022 19:24:13 GMT
ETag: W/"12-7JEJwpG8g89ii7CR/6hhfN27Q+k"
Set-Cookie: route=1663010654.252.42.320963|0e12785838f6d88c8042f8a55371e727; Expires=Wed, 14-Sep-22 19:24:13 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=15724800; includeSubDomains
Via: 1.1 google
X-Powered-By: Express

GET query missing.
i

Ilya Galperin

09/12/2022, 7:25 PM
ya i think the issue here would be pulling
c

Christopher Boyd

09/12/2022, 7:25 PM
then yea, that definitely sounds like dns
✅ 1
i

Ilya Galperin

09/12/2022, 7:25 PM
since it doesn’t seem like we can connect to anything outside the container running the agent
View count: 1