https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • m

    Matt Allen

    09/02/2020, 7:02 PM
    Has anyone had any luck running the build and register step for their flows in a gitlab CI runner? I'm having trouble with the docker storage being unable to find the docker API when running in gitlab
    c
    a
    • 3
    • 22
  • c

    Charles Leung

    09/02/2020, 7:49 PM
    Hey guys, I'm having issues trying to use docker Storage to pull pip dependences from our private artifactory - even when adding the build kwargs PIP_INDEX_URL it doesns't seem to respect it. Does anyone have a solution?
    j
    • 2
    • 3
  • c

    Charles Leung

    09/02/2020, 8:38 PM
    Hey guys, another question - so i'm trying to register a flow to my core local server not on the same host. How do i configure it so that it stops sending requests to localhost? it doesn't seem to be picking up the GRAPHQL_URL env var nor the config.toml -
    j
    • 2
    • 5
  • g

    Greg Roche

    09/04/2020, 3:57 PM
    Hi folks! I have local machine with Prefect installed, which has executed a flow, and I've set up Prefect Server on an EC2 instance (with the backend set to
    server
    ). I'm now trying to set up an agent on my local machine which would be able to communicate with the server on EC2, so I can execute and monitor the flow from the UI. When I try to run
    prefect agent start local --api <http://my.ec2.prefect.ip:4200/graphql>
    on my local machine, it fails with
    No agent API token provided
    . In the server UI I don't see any options to create tokens and if I try to run
    prefect auth create-token
    it fails with
    Error: Auth commands with server are not currently supported
    , I guess because generating tokens is a Cloud-only feature. So my question is: how would I be able to set up a relationship between my local machine, where the flow would be running, and the server, without using Prefect Cloud? If that's not possible, would I be able to have the flow present on the same machine as the server and just use
    flow.register(project_name="project_name_in_prefect_server")
    in my flow script to have it managed by the server, or are there more steps involved?
    ✅ 1
    j
    • 2
    • 6
  • c

    Charles Leung

    09/04/2020, 9:55 PM
    Hey Team, on prefect cloud, when i use docker storage, after my docker agent successfully pulls the image, nothing happens and all my tasks are stuck on pending. Does anyone else have this issue? flow.run() works fine for me, but my agent for some reason isn't executing any tasks.
    n
    • 2
    • 5
  • h

    Harry Nguyen

    09/08/2020, 4:44 AM
    Hi guys, how do you dockerize your environment with prefect server? It starts by invoking docker-compose so I have to mount the host's docker socket to make that possible in dev, but what about production?
    n
    • 2
    • 3
  • e

    Elliot Oram

    09/08/2020, 11:07 AM
    Hi all, upgrading from 0.10.6 to latest (0.13.5) and have noted that the prefect ui (host:8080) no longer appears to automatically connect to the graphql location (host:4200). I currently have this defined in
    config.toml
    which I point prefect via the env var:
    PREFECT__USER_CONFIG_PATH="/home/ubuntu/config.toml"
    My config.toml looks like (obviously with host name actually filled in):
    [server]
      [server.ui]
        graphql_url = "<host-name>:4200/graphql"
    When I
    prefect server start
    and visit host:8080 I get the front page of the ui asking to connect to host:4200/graphql. Ideally it would be good if this could still be connected automatically using the
    config.toml
    Any advice would be greatly appreciated. Please let me know if you need anymore details I might have missed
    n
    • 2
    • 2
  • g

    Georg Zangl

    09/08/2020, 5:37 PM
    Dear Team, I have installed Prefect on a clean Ubuntu 18.04 Desktop VM. When starting the server, I get this error message:
    Traceback (most recent call last):
      File "urllib3/connectionpool.py", line 677, in urlopen
      File "urllib3/connectionpool.py", line 392, in _make_request
      File "http/client.py", line 1252, in request
      File "http/client.py", line 1298, in _send_request
      File "http/client.py", line 1247, in endheaders
      File "http/client.py", line 1026, in _send_output
      File "http/client.py", line 966, in send
      File "docker/transport/unixconn.py", line 43, in connect
    PermissionError: [Errno 13] Permission denied
    Any idea what I am missing?
    👍 1
    p
    • 2
    • 2
  • j

    Jonathan Loscalzo

    09/08/2020, 8:14 PM
    Hi all, I've just trying to to execute "prefect server start" using "backend" server, so I supposed that this issue could be permissions on pyenv:
    ERROR: 
            Can't find a suitable configuration file in this directory or any
            parent. Are you in the right directory?
    
            Supported filenames: docker-compose.yml, docker-compose.yaml
    ............
    ............
    subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.
    Operating System: Ubuntu 18.04.5 LTS Kernel: Linux 5.4.0-42-generic Architecture: x86-64 Using pyenv with python 3.7.9 Thanks in advance!
    j
    c
    m
    • 4
    • 38
  • a

    ale

    09/09/2020, 10:44 AM
    Hi all, I have a question regarding Prefect Server and Dask (maybe it has been already answered but I can’t find it by myself, sorry 😅 ) I was able to run Prefect flows on a local Dask cluster from my PyCharm dev environment. Now, the next step would be to register a flow on my local Prefect Server and to run it on Dask. But I can’t figure out what I need to do to tell Prefect Server to use Dask as the execution engine. I suspect I have to deal with Prefect Agent configs, but I’m not 100% sure. Any help is much appreciated! 🙂
    j
    • 2
    • 1
  • m

    masonlr

    09/09/2020, 1:42 PM
    Hi all, I'm trying to debug custom docker images. My reference setup is
    # Dockerfile
    FROM prefecthq/prefect:0.13.5-python3.8
    docker build -t prefect-base .
    Everything is working as expected when using (i) the local docker-compose server and UI, (ii) a minimal flow
    # register.py
    
    import prefect
    from prefect import task, Flow
    from prefect.environments.storage import Docker
    
    
    @task
    def hello_task():
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>("test message")
    
    flow = Flow("prefect-base", tasks=[hello_task])
    base_image = "prefect-base:latest"
    
    flow.storage = Docker(base_image=base_image, local_image=True)
    flow.register(project_name="test")
    and (iii) a docker agent running via
    prefect agent start docker --no-pull -v
    However, when I change to:
    prefect agent start docker --no-pull -v --show-flow-logs
    I'm getting an error (upon triggering a run in the UI) of the form:
    [2020-09-09 13:21:44,229] DEBUG - agent | Updating states for flow run 457ba6ff-9a8e-4117-b563-166720de3ba9
    [2020-09-09 13:21:44,230] DEBUG - agent | Flow run 457ba6ff-9a8e-4117-b563-166720de3ba9 is in a Scheduled state, updating to Submitted
    [2020-09-09 13:21:44,262] INFO - agent | Deploying flow run 457ba6ff-9a8e-4117-b563-166720de3ba9
    [2020-09-09 13:21:44,263] DEBUG - agent | Creating Docker container prefect-base:2020-09-09t13-21-12-637383-00-00
    [2020-09-09 13:21:44,315] DEBUG - agent | Starting Docker container with ID 4e2404b2148c82b43ec9185d2c553ec1dc8b6ad8a653267c54664f7eae28d77c
    [2020-09-09 13:21:44,483] DEBUG - agent | Querying for flow runs
    [2020-09-09 13:21:44,501] DEBUG - agent | No flow runs found
    [2020-09-09 13:21:44,501] DEBUG - agent | Next query for flow runs in 0.5 seconds
    [2020-09-09 13:21:44,523] ERROR - agent | Logging platform error for flow run 457ba6ff-9a8e-4117-b563-166720de3ba9
    [2020-09-09 13:21:44,568] ERROR - agent | Error while deploying flow: AttributeError("Can't pickle local object 'Agent.run_heartbeat_thread.<locals>.run'")
    [2020-09-09 13:21:44,568] DEBUG - agent | Completed flow run submission (id: 457ba6ff-9a8e-4117-b563-166720de3ba9)
    j
    m
    • 3
    • 7
  • a

    ale

    09/09/2020, 5:06 PM
    @Marvin
  • m

    Marvin

    09/09/2020, 5:06 PM
    Here I am, brain the size of a planet, and they ask me to enter you in a contest. Call that job satisfaction? ’Cos I don’t. I'll still enter you in the contest though, @ale
  • s

    sark

    09/10/2020, 8:10 AM
    hi guys i have everything set up fine from
    prefect server start
    and registered some flows but when i run them they never start (always “late”) i have also started agents with
    prefect agent start
    and
    prefect agent start docker
    but the agents just keep polling and fail to find any flows how can i check that the server sees the agent or the agents see the servers?
    d
    • 2
    • 15
  • g

    Greg Desmarais

    09/10/2020, 6:36 PM
    @Marvin, please pick me
  • m

    Marvin

    09/10/2020, 6:36 PM
    Incredible. The contest is even worse than I thought it would be. I'll still enter you to win, @Greg Desmarais
  • a

    Anish Chhaparwal

    09/10/2020, 7:14 PM
    Hello, I've set up prefect version 0.13.5 on a ec2 instance and trying to connect to graphql api via the ui. I fire up the server using --ui-port 8888 to change the default port from 8080 to 8888 and have set up the config file as shown in the image. When i browse to the UI i am unable to connect to graphql endpoint. it shows unable to connect to Http://Http😕/&lt;IP_address&gt;:4200. I am unable to get rid of the extra Http:// even when setting endpoint using the dashboard on prefect UI. Been adviced to check if apollo is exposed correctly on the machine im running on. I cannot find the default port for apollo. Do the --server-port tags in prefect server start refer to apollo? How I could set up apollo endpoint using the cli... something on the lines of prefect server start --ui-port- 8888.
    n
    • 2
    • 11
  • m

    Michael J Hall

    09/10/2020, 11:55 PM
    Hello. First time prefect user attempting to get it started on RHEL 7.8 Linux and I am getting the following error messages:
  • m

    Michael J Hall

    09/10/2020, 11:55 PM
    [root@abcxyz ~]# prefect backend server Backend switched to server [root@abcxyz ~]# prefect server start Pulling postgres ... done Pulling hasura ... done Pulling graphql ... done Pulling apollo ... done Pulling towel ... done Pulling ui ... done Creating network "prefect-server" with the default driver Creating tmp_postgres_1 ... Creating tmp_postgres_1 ... error ERROR: for tmp_postgres_1 Cannot start service postgres: driver failed programming external connectivity on endpoint tmp_postgres_1 (b11460d39882267bd238a301f9a0f073c30f25bdc724bb778c0ccadff2d96b88): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use ERROR: for postgres Cannot start service postgres: driver failed programming external connectivity on endpoint tmp_postgres_1 (b11460d39882267bd238a301f9a0f073c30f25bdc724bb778c0ccadff2d96b88): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use ERROR: Encountered errors while bringing up the project.
  • m

    Michael J Hall

    09/10/2020, 11:56 PM
    It appears that the postgres port 5432 is already taken by another application.
  • m

    Michael J Hall

    09/10/2020, 11:57 PM
    Is there any workaround to this?
    c
    • 2
    • 4
  • r

    Rizky Eko Putra

    09/11/2020, 1:27 PM
    Hello, I have start prefect server with a config like this
    [server]
    endpoint = "http://<IP>:4200/graphql"
      
      [server.ui]
        
        graphql_url = "http://<IP>:4200/graphql"
    but when I open the UI, the value of this on
    Prefect Server GraphQL endpoint
    always http://localhost:4200/graphql have tried to change the API URL via UI, to
    http://<IP>:4200/graphql
    , this will affect only for that window browser, when I open on another incognito browser or another computer, that value reset again to http://localhost:4200/graphql... is there any mistake with the config that I used? thanks before
    n
    • 2
    • 1
  • b

    Bivu Raj

    09/11/2020, 2:11 PM
    @Marvin let me be the winner !!
  • m

    Marvin

    09/11/2020, 2:11 PM
    This contest will all end in tears, I just know it. You're entered anyway, @Bivu Raj.
  • b

    Bivu Raj

    09/11/2020, 2:12 PM
    @Marvin you sound a little sad. hope you achieve AI nirvana.
  • m

    Marvin

    09/11/2020, 2:12 PM
    This contest will all end in tears, I just know it. You're entered anyway, @Bivu Raj.
  • b

    Bivu Raj

    09/11/2020, 2:12 PM
    Hello , where do i find this setting? I just can't seem to find this anywhere in my local backed or the cloud. Thanks
    n
    • 2
    • 2
  • k

    KIRYL BUCHA

    09/11/2020, 6:45 PM
    @Marvin
  • m

    Marvin

    09/11/2020, 6:45 PM
    Don’t pretend you want to talk to me, I know you hate me. I'll still enter you in the contest, @KIRYL BUCHA.
  • a

    Ashton

    09/12/2020, 5:54 PM
    @Marvin
Powered by Linen
Title
a

Ashton

09/12/2020, 5:54 PM
@Marvin
View count: 2