https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • s

    Shailendra Thakur

    01/31/2021, 11:43 PM
    Hi Team, I am new to prefect & currently got assigned to perform the necessary setup. I am using helm prefect server repo : https://github.com/PrefectHQ/prefect Somehow the agent ( prefect:0.14.5 tag) is not coming up & I am getting "No Tenant found" in pod logs. Can someone please suggest what I am missing here?
    s
    m
    • 3
    • 3
  • p

    Peyton Runyan

    02/01/2021, 1:10 PM
    Hi all, I'm having a bit of trouble getting prefect server to work with github. The environment variable where the agent is running is set
    PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN= <my_token>
    The flow is
    storage = GitHub(
        repo="my_repo",
        path="simple_flow.py",
        secrets=["GITHUB_ACCESS_TOKEN"]
    )
    
    with Flow("github_flow", storage=storage) as f:
    If I remove storage and run it, it runs fine. With storage, it hangs indefinitely on
    scheduled
    . Any ideas?
    k
    • 2
    • 6
  • a

    Anton Rasmussen

    02/01/2021, 1:45 PM
    Hi Prefect community, I'm having trouble getting the Docker Agent working when deployed in a Docker container. I have registered a flow with Docker storage in the cloud. I'm building the Docker image in a slightly non-standard way, but the flow works fine as long as I execute it though a Docker agent running on my laptop (i.e. not in a docker container). Next, I try to deploy an agent using docker stack and a simple docker-compose.yml, but this seems unable to execute the same flow correctly. Here is what I see: - There is connection to cloud, agent says: "Found 1 flow run(s) to submit for execution." - There is connection to the private registry where flow is stored, agent says: "Successfully pulled image *****/template-flow:dbea2161" - I think flow container works, agent says: " Docker container ***** started" - Strangely: - agent reports job complete (?): "Completed flow run submission (id: a507921f-b9c6-400a-943f-a984b99eadbc)" - cloud UI shows both tasks for that flow run as pending. If I enable " --show-flow-logs" for the Docker-agent-in-Docker, the agent prints below traceback (after above list of output). So my guess is that something inside my flow container is trying to talk to the cloud API, but somehow failing due to missing credentials. I don't understand why it's failing in the Docker-agent-in-Docker case and not in the Docker-agent-from-terminal case. Any hints appreciated.
    Traceback (most recent call last):
       File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 451, in _request
         json_resp = response.json()
       File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 900, in json
         return complexjson.loads(self.text, **kwargs)
       File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
         return _default_decoder.decode(s)
       File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
       File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
         raise JSONDecodeError("Expecting value", s, err.value) from None
     json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
     
     The above exception was the direct cause of the following exception:
     
     Traceback (most recent call last):
       File "/usr/local/bin/prefect", line 8, in <module>
         sys.exit(cli())
       File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
         return self.main(*args, **kwargs)
       File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
         rv = self.invoke(ctx)
       File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
         return _process_result(sub_ctx.command.invoke(sub_ctx))
       File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
         return _process_result(sub_ctx.command.invoke(sub_ctx))
       File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
         return ctx.invoke(self.callback, **ctx.params)
       File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
         return callback(*args, **kwargs)
       File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 49, in flow_run
         result = client.graphql(query)
       File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 298, in graphql
         result = <http://self.post|self.post>(
       File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 213, in post
         response = self._request(
       File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 454, in _request
         raise ClientError(
     prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
    k
    • 2
    • 6
  • p

    Peter Roelants

    02/02/2021, 10:25 AM
    How can I set the logging level of the prefect graphql service? I tried adding both
    PREFECT_SERVER__LOGGING__LEVEL: WARN
    and
    PREFECT__LOGGING__LEVEL: WARN
    environment variables to the docker-compose file, however I still see
    INFO
    log messages.
    a
    • 2
    • 2
  • s

    Scott Jones

    02/02/2021, 6:46 PM
    Is it possible to create your own executor, esp., one in a different language? I am concerned about how Python centric everything looks - I’m looking for the ability to run both Python and Julia tasks (the Julia tasks very well might be calling into Python via PyCall, so I’d need to make sure that that could be handled also)
    a
    j
    • 3
    • 5
  • r

    Rui Rego

    02/02/2021, 8:10 PM
    Hi. I'm new to prefect (and docker by the way). I'm able to start prefect server via terminal inside Pycharm. I'm starting a local agent from a terminal in Pycharm also. Then I'm registering a flow. I open the UI and the flow is correctly registered and scheduled. I'm letting the flow run for a few times and it executes successfully. Then I close Pycharm and that terminates the terminals where I have started prefect server and the agent. If I'm understanding it well, prefect server is still running on a docker container (I can see that via Docker Desktop) but I need to start a new agent. So I run a python script to start the agent and it is successful:
    from prefect.agent.local import LocalAgent
    LocalAgent().start()
    After that, all scheduled executions of the flow are failing with:
    Failed to load and execute Flow's environment: StorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'src\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
    What am I doing wrong? I am using Windows 10, python 3.9 and a virtual environment (venv).
    • 1
    • 1
  • b

    BK Lau

    02/02/2021, 10:30 PM
    Q: Is the Prefect server multitenant i.e. multi users with different logins can submit workflow thru a single server instance??
    j
    • 2
    • 2
  • d

    David Kuda

    02/03/2021, 4:42 PM
    Hello guys, I just did something really silly, I ran into an infinite loop which triggered all flows to run. Luckily I could stop that. However, my run history is somewhat spoiled now (see picture). Any way I can reset the run history?
    g
    • 2
    • 3
  • s

    Sean Talia

    02/03/2021, 5:49 PM
    has anyone ever run into this issue before? i was running
    prefect server start
    just fine the other day, and in the last 10-15 minutes when I've tried running it, i'm getting sporadic errors for the various docker compose services that need to be pulled. one time i might get
    ERROR: for apollo unauthorized: authentication required
    , another time i get
    ERROR: for postgres unauthorized: authentication required
    , a 3rd time i get the same for
    graphql
    , or some combination of the above
    j
    • 2
    • 2
  • j

    josh

    02/03/2021, 6:25 PM
    If anyone is currently experiencing authentication/authorization issues with pulling the Prefect server images it is likely due to the current DockerHub service interruptions where the images are being hosted https://status.docker.com/pages/533c6539221ae15e3f000031
    • 1
    • 1
  • n

    Nikul

    02/03/2021, 6:38 PM
    Hi all, I'm trying to use a remote Kubernetes cluster to do my computations, but submit them directly from a local machine without having to first register a flow, and after the flow has run I want to avoid having to collect the results, eg from GCS (so I would like the computations to be interactive). Is this possible with Prefect? Thank you (The KubernetesAgent and Prefect Server are both running in-cluster).
    j
    • 2
    • 3
  • j

    Josh Greenhalgh

    02/03/2021, 8:02 PM
    Is there anyway to batch mapped tasks? Specifically I have a flow that will kick of many runs of another parametrised run but I need to minimise the number of concurrent runs
    m
    • 2
    • 20
  • k

    kevin

    02/03/2021, 10:47 PM
    does a prefect task have any knowledge of what project context it's being executed in?
    j
    • 2
    • 11
  • t

    Timo

    02/04/2021, 8:21 AM
    Hi there, I've got a question about
    flow.register( idempotency_key=flow.serialized_hash(),...)
    . If I register my flows with this method, then, if I change the code of a single task, it creates no new version. Only if I make a change at the flow itself. Is it possible to make a new version if the code of the task changes (but not every time, like without idempotency_key)? Thanks
    m
    • 2
    • 2
  • b

    Bob Primusanto

    02/04/2021, 9:49 AM
    Hello fellow prefect user, have you guys ever experienced this type of error?
    Unexpected error: AttributeError("'StartFlowRun' object has no attribute 'run_config'")
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 867, in get_task_run_state
        logger=self.logger,
      File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 298, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 453, in method
        kwargs.setdefault(attr, getattr(self, attr))
    AttributeError: 'StartFlowRun' object has no attribute 'run_config'
    a
    d
    • 3
    • 3
  • m

    Michael Hadorn

    02/04/2021, 10:57 AM
    Hi there :) My Problem is: the prefect agent on the server doesn't deploy my flow run. (he's only waiting for flow runs) In the UI the flow is registered and scheduled, but the agent does not execute it. I see the flow if I do: prefect get flow-runs
    > prefect get flow-runs
    NAME FLOW NAME STATE AGE START TIME ID
    spiffy-scorpion Load Scheduled 13 minutes ago 17a13e07-d217-4a10-a821-5947657a1a48
    I use the local agent. In the UI can see the agent and also if I stop the UI, the agent complains. Also the prefect server start env is used for the whole setup. Would be very amazing to get some help. Thanks
    g
    • 2
    • 2
  • r

    Rui Rego

    02/04/2021, 2:57 PM
    Hi. I have a flow that passes a results object (the results of session.query - sqlalchemy) between tasks. If I run the flow with flow.run() it executes ok. If I run it from the UI it fails with this error:
    Unexpected error: TypeError("cannot pickle 'weakref' object")
    Traceback (most recent call last):
      File "d:\302_investing\venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "d:\302_investing\venv\lib\site-packages\prefect\engine\task_runner.py", line 891, in get_task_run_state
        result = self.result.write(value, **formatting_kwargs)
      File "d:\302_investing\venv\lib\site-packages\prefect\engine\results\local_result.py", line 116, in write
        value = self.serializer.serialize(new.value)
      File "d:\302_investing\venv\lib\site-packages\prefect\engine\serializers.py", line 70, in serialize
        return cloudpickle.dumps(value)
      File "d:\302_investing\venv\lib\site-packages\cloudpickle\cloudpickle_fast.py", line 73, in dumps
        cp.dump(obj)
      File "d:\302_investing\venv\lib\site-packages\cloudpickle\cloudpickle_fast.py", line 563, in dump
        return Pickler.dump(self, obj)
    TypeError: cannot pickle 'weakref' object
    Why the different behavior between Core and Server?
    ✅ 1
    c
    m
    • 3
    • 4
  • s

    Samuel Hinton

    02/04/2021, 3:35 PM
    Has anyone successfully launched an agent using docker-compose that talks to the apollo server? Screenshotting my current agent container in case theres an obvious error - currently it runs and connects but maybe the config is messed up because it never finds any flows to run. If I run
    prefect local agent start --label any
    in my terminal, it works beautifully. (Edit: Actually now when I run it its stuck on “Waiting for flow runs”). Inside a docker container (using local or docker agents) - connects fine by then always says “No flow runs found”. This is the same whether I run a docker or local agent in the docker-compose, with and without attaching labels to anything, with both local storage and S3, and with local, universal and docker run configs explicitly set.
    m
    • 2
    • 8
  • j

    Joseph

    02/04/2021, 6:27 PM
    I am running a very simple flow with a custom executor (I have an executor that is similar to Dask for a very large private compute cluster). When I run the flow locally (
    flow.run()
    ) it successfully completes and using diagnostic tools for my cluster I can see the jobs complete. I can also register the flow with the server (
    flow.register
    ). But when I quick run the flow, from the UI’s perspective it looks as if it never completes (it is stuck in the “running” state). Using my cluster diagnostics I can see the actual jobs get scheduled and complete. So I’d guess whatever sort of events are meant to be sent to the server in order to indicate state changes in the jobs are not being sent? How can I diagnose what’s happening?
    s
    • 2
    • 7
  • n

    Neeraj Vyas

    02/05/2021, 8:50 AM
    Hi everyone, I am new to prefect, when using kubernetes agent, tasks got stuck at scheduled state. Same works fine with local agent.
    n
    • 2
    • 1
  • j

    Joao Erik Melo

    02/05/2021, 8:27 PM
    Hi there, I'm new in Prefect and I'm facing some issues in an Flow I need to implement. I need a Flow with few tasks in sequence (Task1 -> Task2 -> Task3). But depending on diferent conditions each Task can be SKIPPED according with its respective condition. I need each task (that are not skipped) process a dataset and pass this data precessed to the next NOT SKIPPED task. There a two main issues: 1. If Task1 and Task3 a forced to run, and Task2, in the middle, forced to be SKIPPED, the downstream Task3 is SKIPPED too (if skip_on_upstream_skip=True). Although, if I set skip_on_upstream_skip to False (in all 3 tasks), Task2 is NOT skipped as should be [image].  2. How can I keep the data flow through a sequence of tasks, if one of then is skipped? When a task is skipped, I lost the data flow through it. Woudl be the best practice use a shared path for all tasks (a hive table that is read and overwrited for each task, for example)  is OR there is a more Prefectonic way to do that? May someone help me, please?
    with Flow("conditional-branches") as flow:
    
        cond1 = check_condition1()
        with case(cond1, True):
            val1 = task1()
    
        cond2 = check_condition2()
        with case(cond2, True):
            val2 = task2()
    
        cond3 = check_condition3()
        with case(cond3, True):
            val3 = task3()
    
    
    
        flow.set_dependencies(
        task=val1,
        downstream_tasks=[val2])
    
        flow.set_dependencies(
        task=val2,
        downstream_tasks=[val3])
    n
    • 2
    • 11
  • s

    Sagun Garg

    02/08/2021, 4:29 AM
    Hi Everyone, I am sure I am making a small mistake here somewhere. After I run creating the tenant command
    $ prefect backend server && prefect server create-tenant --name default --slug default
    This is the error I get on my CLI
    raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'message': '[Errno -2] Name or service not known', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_tenant'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '[Errno -2] Name or service not known'}}}]
    I am getting this error in my prefect-graphql container
    2021-02-07 01:57:13[Errno -2] Name or service not known7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:13GraphQL request:2:37e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:131 | mutation ($input: create_tenant_input!) {7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:132 | create_tenant(input: $input) {7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:13| ^7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:133 | id7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:13Traceback (most recent call last):7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:13File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:13return await result7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:13File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve7e86527e64b44aa992ab290af4938b13
    2021-02-07 01:57:13result = await result7e86527e64b44aa992ab290af4938b13
    I have setup prefect-server on AWS Fargate ECS using the docker-compose.yml as a guidleine to create my own task definition where I am doing a multi-container setup with ENV defined in Key, Values of each container. I think there is some issue in ENV Hasura Admin Secret in prefect-graphql, prefect-apollo as I see some errors in headers while the tenant is being created. So I have removed the Hasura Admin password from my prefect-hasura container. I am still getting the above errors. ANY HELP IN THIS REGARD WILL BE APPRECIATED...I THINK I AM 1 STEP AWAY FROM SEEING PREFECT-SERVER RUN AGAIN 🙂 I HAD IT RUNNING BEFORE BUT HIT THIS ISSUE.
    j
    • 2
    • 1
  • m

    Mathijs Schoorl

    02/08/2021, 1:00 PM
    Did anyone else noticed the issue that a KubernetesAgent is labeled as unhealthy in the UI but really isn't? It happens when the agent loses it's connection with the server, reconnects after a while but in the UI still shows as an unhealthy agent. It is picking up flows so the agent does work as expected. The UI just doesn't reflect the state correct. If I restart the pod where the agent runs on and it connects after this restart the agents is showing as healthy again. I'm running Prefect server version 0.14.3. Or might this be something in my configuration? We have only 1 agent running.
    j
    • 2
    • 2
  • c

    Chris Goddard

    02/08/2021, 2:12 PM
    I'm having a pretty buggy experience on Prefect Cloud lately - the page will hang/freeze and now often redirect to cloud.prefect.io/404
    n
    • 2
    • 2
  • d

    Dana Merrick

    02/08/2021, 4:39 PM
    I'm having trouble connecting to graphql via the Prefect Server webUI. I can reach it on localhost:4200, <ip>:4200, and <fully-qualified-name>:4200, but I always get
    Oops! It looks like something went wrong when trying to connect; make sure Prefect Server is running at the URL above and try again.
    k
    • 2
    • 20
  • e

    Edward Chen

    02/09/2021, 9:15 AM
    Hi all, I’m having trouble running the hello-world flow based on the tutorial (First Flow). It’s been modified and runs on all my coworkers computers but for some reason, when I run it, I get the
    Name or service not known
    error message. I confirmed that GraphQL is not running based on the error message on the top right of the prefect UI. Stack trace:
    apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
    apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
    graphql_1   | 
    graphql_1   | Running Alembic migrations...
    graphql_1   | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
    graphql_1   | INFO  [alembic.runtime.migration] Will assume transactional DDL.
    graphql_1   | 
    graphql_1   | Could not upgrade the database!
    graphql_1   | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x40133ab7d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
    Based on the stack trace, it confirms that there is an issue with graphql, I’m just not sure how to dive deeper. I have tried Dagster and initially came across some issues with compatibility with the M1 chip so I created an EC2 instance and tried the prefect first flow tutorial in that one as well. I hit an issue with docker compose not being found so rather than splitting into two separate paths, I’m going to try to focus on getting the first flow tutorial working on my local instance (macos)
    m
    • 2
    • 11
  • j

    Joël Luijmes

    02/09/2021, 11:02 AM
    What happens if I update Prefect (or restart it) while running a flow (specifically Kubernetes Agent)? Will the new agent correctly monitor the flow or could it introduce undefined behhavior?
    ✅ 1
    m
    • 2
    • 6
  • b

    Ben Epstein

    02/09/2021, 10:34 PM
    I’m having trouble finding this in the docs - is there a way to specify package dependencies for a run? I see you can use a kubernetesExecutor like airflow and specify a docker image, but that would involve rebuilding your image every time dependencies change. Is there a way to include a
    requirements.txt
    or
    conda.yaml
    with a job? Thanks!
    j
    • 2
    • 15
  • m

    Michael Hadorn

    02/10/2021, 7:49 PM
    Hi there 🙂 One question about dynamic flows (build at runtime) and the prefect server/cloud. Our ETL process contains entities to load (stored in the database). The list of these entities can change over time (because of development). How can I achieve a dynamic load, which will load all entities existing at runtime and be able to manage this via the Server/Cloud? I can not use the task-mapping function, because the entities have own dependencies (also stored in the database) and these will be dynamically resolved at runtime. Is there a way to solve this kind of problem with prefect/UI? I developed this with the localDaskRun and the explicit run-call. Then its working, because the flow is built everytime. Now via Server, I got errors when the task is changing:
    KeyError: 'Task slug XX not found in the current Flow; this is usually caused by changing the Flow without reregistering it with the Prefect API.'
    Best Michael
    j
    • 2
    • 7
  • s

    Si Wong

    02/11/2021, 8:00 AM
    Hi all, I am looking to use the helm chart to deploy on a AWS EKS cluster. Having little knowledge and spending some time reading through the docs, I was wondering if anyone else has done this and have any notes they can send me so I can attempt to set this up today? Thanks in advance, any advice would be appreciated!
    m
    j
    • 3
    • 19
Powered by Linen
Title
s

Si Wong

02/11/2021, 8:00 AM
Hi all, I am looking to use the helm chart to deploy on a AWS EKS cluster. Having little knowledge and spending some time reading through the docs, I was wondering if anyone else has done this and have any notes they can send me so I can attempt to set this up today? Thanks in advance, any advice would be appreciated!
m

Michael Adkins

02/11/2021, 4:07 PM
Hi @Si Wong -- I maintain the helm chart, let me know if you have any specific questions. It also may be worth looking at the IAC in https://github.com/PrefectHQ/server/pull/171 -- it's just for Azure/GCP right now but it can give you an idea of what infrastructure needs to be deployed.
s

Si Wong

02/11/2021, 4:13 PM
Thank you for your response @Michael Adkins, I will take a look and if it's ok with you, can I reach out over slack?
m

Michael Adkins

02/11/2021, 4:14 PM
Feel free to reach out in this thread, we prefer that users do not DM us here 🙂
👌 1
Those IAC examples are more complicated because they're deploy a separate postgres instance which I'd recommend doing for production but for just getting started all you really need is a small EKS cluster and the helm command.
s

Si Wong

03/09/2021, 4:30 PM
Thanks @Michael Adkins, I've revisited this and managed to get the deployment of Prefect-Server working on an EKS cluster. I also took your advice and deployed a Postgres (RDS) instance and configured a k8s secret to hold the database password. All of that works out the box, thank you! I have ran into another challenge and it's most likely due to my k8s technical debt in some key areas but I have now modified the helm chart by including an ingress template and it looks to work both for the UI and Apollo containers. I changed the service from loadbalancer to cluster IP and using the nginx annotation in place of. I have now enabled the agent and tenant but these containers seem to run in an error loop (see attachment). I've narrowed it down to these Environment variables and not quite sure how to link them locally and would prefer not to hardcode the dns in here. Do you have any ideas how this should be configured?
PREFECT__CLOUD__API: <http://prefect-server-apollo.prefect-harbor:4200/graphql/>
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='prefect-server-apollo.prefect-harbor', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f88206e0250>, 'Connection to prefect-server-apollo.prefect-harbor timed out. (connect timeout=15)'
m

Michael Adkins

03/09/2021, 4:41 PM
Hmm. So that url is constructed to use the builtin cluster DNS which is
<service-name>-<namespace>
which should be accessible although I'm not sure how that changes once an ingress is added.
Have you overridden these values? https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields
I wonder if looking up the dns names would help?
kubectl exec -i -t dnsutils -- nslookup <service-name>.<namespace>
s

Si Wong

03/09/2021, 5:01 PM
Thanks @Michael Adkins I will take a look through this, it's nice to be pointed in the right direction so much appreciation for that. Just a quick question re: the env variable above, does it matter that it's labeled
PREFECT__CLOUD__API
despite using prefect-server?
m

Michael Adkins

03/09/2021, 5:14 PM
Yeah
CLOUD
just means "backend api" there 🙂
👌 1
s

Si Wong

03/09/2021, 5:23 PM
@Michael Adkins your link helped me understand what needed to be done. I was able to navigate to the
_helpers.tpl
file under the apollo directory for the chart and modify out the port. As I have setup a proxy using nginx. I changed it to the below snippet and it now works as expected with a tenant and agent. Onwards and upwards
{{- define "prefect-server.apollo-api-url" -}}
{{- $host := include "prefect-server.apollo-hostname" . -}}
{{- $port := "4200" -}}
{{ printf "<http://%s>" $host }}
{{- end -}}
m

Michael Adkins

03/09/2021, 5:25 PM
Ah that makes sense! If you want to contribute back there is an open issue to add ingress support out of the box :) if you don't have time to finish it perhaps someone else will if you open a PR with your changes
👀 1
s

Si Wong

03/09/2021, 5:25 PM
I've never actually opened a PR before 😐
m

Michael Adkins

03/09/2021, 8:25 PM
If you want to send me a DM I'd be happy to walk you through it, otherwise nbd 🙂
s

Si Wong

03/15/2021, 4:49 PM
Hi @Michael Adkins, me again and apologies! So I have an interesting challenge I am trying to solve and looking for advice before I run down a rabbit hole! I have authentication wrapped around the UI but would like to achieve the same setup for the Apollo endpoint. However, if I wrap auth redirect on the Apollo ingress, the UI can no longer connect due to the headers. For some reason I can't reference the Apollo [apolloApiUrl: friendly.dns.com] as a private kubneretes dns variable i.e.
<http://prefect-server-apollo.prefect.svc.cluster.local:80/graphql>
It needs to be a public domain which we can't protect, can you advise how I could potentially link the UI > Apollo endpoint utilising kubernetes backbone network rather than the browser looking for a public facing DNS?
m

Michael Adkins

03/15/2021, 5:09 PM
Hey! I don't really have any good advice for you. I'm not an expert on auth patterns in K8s. I presume the private dns variable is not working because the UI running in your user's browser needs to be able to contact the API directly (https://github.com/PrefectHQ/server/issues/212). Generally, Server isn't intended to handle auth and most things you can add will be top-level whereas Cloud has auth all the way down to the database and uses Okta for SSO. Having the UI proxy requests to the API is feasible but would be a significant amount of refactoring in the UI which I don't think our team has the bandwidth for.
j

Jonathan Wright

03/16/2021, 11:31 AM
Hi @Michael Adkins thank you for all the great information. I work with @Si Wong. You mentioned above that Cloud uses Okta for SSO. We have our own Okta domain. Which makes me think could we configure Prefect UI to point at our Okta domain? So that traffic from the UI to Apollo would have an authorization header attached. I can see the config used for Cloud here that we would need to change? https://github.com/PrefectHQ/ui/blob/master/.env Can you see any unforeseen issues if we attempt this? Thank you again.
m

Michael Adkins

03/16/2021, 3:01 PM
Hey @Jonathan Wright -- I don't work on the UI but I'll ping someone on that team.
View count: 3