https://prefect.io logo
Join Slack
Channels
ask-community
announcements
ask-marvin
best-practices
data-ecosystem
data-tricks-and-tips
events
feedback-deployment-concurrency
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
intros-test
livestream-chatter
marvin-ai
marvin-in-the-wild
pacc-apr-10-11-2024
pacc-apr-30-may-1-2024
pacc-apr-9-10-2025
pacc-aug-16-2023
pacc-aug-22-23-2023
pacc-aug-28-29-2024
pacc-aug-29-30-2023
pacc-clearcover-june-12-2023
pacc-dec-17-18-2024
pacc-feb-13-14-2024
pacc-feb-26-27-2025
pacc-jan-28-29-2025
pacc-july-11-12-2023
pacc-july-17-18-2023
pacc-july-30-31-2024
pacc-july-6-2023
pacc-june-14-2023
pacc-june-20-21-2024
pacc-london-2023
pacc-london-sept-2024
pacc-mar-12-13-2024
pacc-may-31-2023
pacc-nov-19-20-2024
pacc-nov-8-2023
pacc-nyc-may-2024
pacc-oct-11-12-2023
pacc-oct-1-2-2024
pacc-oct-16-17-2024
pacc-sept-13-14-2023
pacc-sept-20-21-2023
pacc-sept-26-27-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-contributors-archived
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-and-tell
Powered by
# prefect-server
  • m

    matt forbes

    09/18/2020, 9:02 PM
    I've got prefect server and all its components deployed and talking to each other, now trying to interact w/ it from the cli and running into 400 errors trying to do the first few steps (create tenant)
    n
    • 2
    • 25
  • m

    matt forbes

    09/18/2020, 11:56 PM
    this issue to override the default apollo url was closed a few days ago but I am still unable to get this to default to anything other than localhost: https://github.com/PrefectHQ/ui/issues/99
    n
    • 2
    • 49
  • j

    Jovan Visnjic

    09/21/2020, 1:54 PM
    Hello everyone! I have a question about having an option of running the same workflow both with the prefect server and without it. For example, when we just want to evaluate the correctness of the workflow and write some unit tests for it, we run it in a console without the server, using
    LocalResult
    . But then we'd like to register the same workflow on the server and run it there on a large scale, using
    PrefectResult
    and cache to the database. We're struggling to get the
    LocalResult
    and
    PrefectResult
    behave in the same way, so we could just switch between one and the other. Does this make sense at all - to have a switch between offline and server run? Or should one result class handle all cases, perhaps a custom one?
    k
    • 2
    • 5
  • j

    Julian

    09/21/2020, 5:15 PM
    Hey, I want to contribute to an open pull Request https://github.com/PrefectHQ/prefect/pull/3330 The production Code seems fine to me and the core maintainers as well. However, due to a failed unittest this branch cannto be merged into master yet, so I wanted to fix this unittest. How should I push the Unittest-changes into this branch, since I want this feature as soon as possible. Do I have to create a new Branch (with identical changes + testfix) or can I somehow correct the Test and it will be merged soon?
    k
    • 2
    • 1
  • m

    mot s

    09/22/2020, 8:01 PM
    Hi Prefect Community, just started going over prefect docs, blogs and videos. We are a big on-premise data company (in a highly regulated environment) and trying to see if prefect is a good solution for us. We are a big pyspark shop on CDH (Cloudera hadoop distro) and using ML (h2o). Looking at prefect to help us with mainly 2 things: 1. document data and state dependencies across our pyspark code, orchestrate pyspark based flows. Reading up on prefect, it seems like it can do these 2 things but a little lost on architecture of how this can be deployed. Basic questions - would prefect agent be running on the edge (hadoop gateway servers) and prefect backend server on K8s cluster? How would flow orchestration work? Would agent be kerberos aware? Sorry for lot of these basic questions - could not find info on prefect deployment in an on-prem setup with spark being the execution engine.
    j
    • 2
    • 2
  • s

    sark

    09/23/2020, 9:07 AM
    hi guys i have an empty flow
    Copy code
    with Flow('flow', storage=docker_storage) as flow:
        passs
    using docker storage
    Copy code
    docker_storage=Docker(registry_url="<http://gcr.io/repo|gcr.io/repo>", dockerfile='Dockerfile')
    where the dockerfile is just
    Copy code
    FROM prefecthq/prefect:all_extras-0.13.7
    and the agent logs show
    Copy code
    [2020-09-23 09:01:30,367] INFO - agent | Pulling image <http://gcr.io/repo/flow:2020-09-23t08-59-44-707304-00-00|gcr.io/repo/flow:2020-09-23t08-59-44-707304-00-00>...
    ERROR: gcloud failed to load: No module named '_sqlite3'
        gcloud_main = _import_gcloud_main()
        import googlecloudsdk.gcloud_main
        from googlecloudsdk.api_lib.iamcredentials import util as iamcred_util
        from googlecloudsdk.core.credentials import transports
        from googlecloudsdk.core.credentials import http
        from googlecloudsdk.core.credentials import creds as core_creds
        import sqlite3
        from sqlite3.dbapi2 import *
        from _sqlite3 import *
    
    This usually indicates corruption in your gcloud installation or problems with your Python interpreter.
    
    Please verify that the following is the path to a working Python 2.7 or 3.5+ executable:
        /opt/prefect/.local/share/virtualenvs/prefect-Ta6MxEs8/bin/python3
    
    If it is not, please set the CLOUDSDK_PYTHON environment variable to point to a working Python 2.7 or 3.5+ executable.
    
    If you are still experiencing problems, please reinstall the Cloud SDK using the instructions here:
        <https://cloud.google.com/sdk/>
    so this seems like it is the agent not having proper GCR auth? how can i configure the docker agent for this? is there a way to configure the image of the agent like it is possible to configure the image for the flow?
    j
    • 2
    • 4
  • g

    Greg

    09/23/2020, 3:19 PM
    Hi ! I've updated server, now all tasks running for more than 15min are killed by the ZombieKiller ! How can I disable it ?
    j
    • 2
    • 10
  • g

    Greg

    09/23/2020, 3:40 PM
    I reverted back to 0.13.4 ...
  • c

    charmatski

    09/25/2020, 2:48 PM
    Hi, I would like some help understanding the utility of the Docker image (https://hub.docker.com/r/prefecthq/prefect) I thought I could use this and run it to get a prefect environment running locally.. but trying to start the prefect server I get
    No such file or directory: 'docker-compose': 'docker-compose'
    Should docker-compose not be installed inside the image and this be a read-to go dev environment?
    ✅ 1
    n
    s
    • 3
    • 7
  • g

    Gitanshu Sardana

    09/25/2020, 10:54 PM
    Hi. I just updated to prefect 0.13.8 and I'm seeing logs for graphql foreign key violations on starting the server. I cleaned up the existing docker images and volumes, and everything in ~/.prefect except config.toml and backend.toml, just to see if that changes anything, but I still see it.
    n
    m
    • 3
    • 19
  • g

    Gitanshu Sardana

    09/25/2020, 10:55 PM
    Untitled
  • v

    Vipul

    09/27/2020, 10:39 AM
    Hi, I was trying to add a new cloud hook on Prefect Core for Web with URL as http://127.0.0.1:5000/alert to test this feature but it keeps complaining that it is invalid URL. Is it possible to test this feature in development environment on local host? Thanks
    n
    • 2
    • 3
  • a

    ale

    09/28/2020, 3:01 PM
    Hi all, we’re trying to figure out how to build the image of a Prefect Flow inside a Jenkins pipeline. We would want to avoid using Prefect directly to build the image, mainly because we don’t want to have Jenkins to depend on Prefect for building docker containers. Have you find yourself in such situation before? How did you manage to create Prefect docker images without relying on storage.build() and flow.register()? Do you think it is something doable?
    c
    • 2
    • 2
  • s

    Steve Taylor

    09/28/2020, 8:30 PM
    Design question. Why did you choose to front Hasura with Apollo instead of A) the other way around or B) no Apollo at all, and just use Hasura and APIs? I've got a decision to make about a platform, and I'm amazed by Hasura. Can I bother you for a few moments of your time as to how Apollo ended up in the mix? Clearly a low priority, but I'm awfully curious. Thanks!
    c
    • 2
    • 3
  • h

    Hagai Arad

    09/29/2020, 2:29 PM
    Hi, I have a master flow which triggers several child flows (using FlowRunTask). When looking at the logs of a specific run of the master flow I can see it triggered the child flows, but I can’t get from there to the child logs. Is there a way to enable it? In other words, I want to know if a flow was triggered by another flow or by itself. Is there a way to know that using the UI? Thanks!
    j
    a
    • 3
    • 8
  • a

    Ana

    10/01/2020, 1:50 PM
    Hello guys, I'm very happy to be heard. I have a simple question: does Prefect run on Spark Cluster? Because I didn't find that in a clear explanation and all the documentation is oriented Dask oriented. Can someone explain this to me? Thanks
    n
    j
    • 3
    • 13
  • m

    Matt Allen

    10/01/2020, 3:57 PM
    Is there somewhere I can find more info about task health checks? I've got some flows failing because of them and it's not clear what might actually cause that. Are these healthchecks run in a thread from the flow process or something like that?
    c
    • 2
    • 11
  • c

    Charles Leung

    10/01/2020, 5:43 PM
    Hey Team! I see that the nomad-agent was deprecated; is there a particular reason this was the case?
    c
    • 2
    • 13
  • e

    Emma Willemsma

    10/01/2020, 8:15 PM
    Hey everyone, my teammate @Zaid Naji and I are trying to run flows using the Fargate agent. We're getting this error when we try to run a flow:
    Copy code
    An error occurred (InvalidParameterException) when calling the RegisterTaskDefinition operation: Invalid 'cpu' setting for task.
    The entrypoint we're using for the agent is:
    Copy code
    ["prefect","agent","start","fargate","cpu=256","memory=1024","networkConfiguration=$NETWORK_CONFIGURATION"],
    Does anyone know what we should we be setting for
    cpu
    to get it working?
    n
    d
    +4
    • 7
    • 49
  • l

    Luke

    10/01/2020, 10:25 PM
    Hey, a basic stuff but wanted to confirm.. to be able to see and run the flow in my own infrastructure I need to register the flow in the cloud account? No way to use it without the account? I ran prefect server but cannot force to register flow to local instance
    n
    • 2
    • 8
  • c

    Charles Leung

    10/02/2020, 7:00 PM
    Hey Team! If i were running prefect server locally (not on cloud), how would i swap out the docker image of postgres with a postgres i already have running? Would it be a change in the following for docker-compose.yml: PREFECT_SERVER_DB_CMD: ${PREFECT_SERVER_DB_CMD:-"echo 'DATABASE MIGRATIONS SKIPPED'"} PREFECT_SERVER__DATABASE__CONNECTION_URL: ${DB_CONNECTION_URL} would this be recommended? or you think prefect should have its own dedicated postgres instance?
    👀 1
    j
    • 2
    • 3
  • b

    Bruce Haggerty

    10/04/2020, 7:06 PM
    Hi, apologies for a super basic question but I can't seem to get prefect server to recognize the value of "server.ui.apollo_url" ... the line exists in the config.toml file, but the dashboard still shows it as "localhost" and suggests that the value isn't set. It seems to be the same issues as https://stackoverflow.com/questions/63719945/how-to-configure-graphql-url-in-prefect-server-0-13-5 but I just installed this today, and that issue is marked as resolved
    j
    • 2
    • 3
  • a

    Anish Chhaparwal

    10/05/2020, 5:50 PM
    hey, I want to deploy my flow using docker. I'm using the code from prefect docs : Flow with docker (first image attached). I do not push it to any registry as i want to run the docker image on my local machine. After starting the docker agent using
    Copy code
    prefect agent start docker
    i am able to register the flow to the UI but it always remains in pending state and never gets executed(refer images attached). Can someone explain where i am going wrong?
    k
    • 2
    • 10
  • d

    Darren Forsdike

    10/07/2020, 9:33 AM
    Hi, has anyone example code of a flow to run a docker image on a schedule ?
    j
    • 2
    • 1
  • n

    Newskooler

    10/07/2020, 1:08 PM
    Hi 👋 If anyone has experience starting
    prefect server start
    via a running docker image, can you please help me fix an error I keep getting : / I summarized it here: https://github.com/PrefectHQ/prefect/discussions/3449
    c
    • 2
    • 3
  • t

    tom

    10/08/2020, 5:27 PM
    Hi! I have Prefect server & UI running locally (
    prefect server start
    ) and my flows are executing via the agent (
    prefect agent start
    ). I was wondering how I could get them to run on Dask? I have
    dask-scheduler
    and a few
    dask-worker
    processes running. When I run my flow using
    flow.run(executor=DaskExecutor(address="<tcp://xxx:8786>"))
    everything works well, but when I run the flow via prefect server’s UI it doesn’t run on Dask. What am I missing? I tried running the server and agent with
    export PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS="prefect.engine.executors.DaskExecutor"
    and
    export PREFECT__ENGINE__EXECUTOR__DASK__ADDRESS="<tcp://xxx:8786>"
    set but this didn’t help.
    c
    s
    • 3
    • 3
  • d

    DJ Erraballi

    10/08/2020, 10:41 PM
    Other than running scheudling for the flows and acting as the api backbone for the prefect UI; the major roles that prefect server performs is to: 1. checkpoint tasks?
  • d

    DJ Erraballi

    10/08/2020, 10:42 PM
    attempting to understand the scalability differences for single node prefect server vs prefect cloud
  • d

    DJ Erraballi

    10/08/2020, 10:42 PM
    is the major scaling problem for concurrent flows, that the prefect server needs to be available to recieve checkpointing data and store it?
    c
    • 2
    • 13
  • j

    J.R. Carneiro

    10/08/2020, 11:02 PM
    Hi, I am having a dependency issue and I am unsure whether anyone has encountered this before
    n
    • 2
    • 6
1...456...51Latest