https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • d

    Dipen Chawla

    08/24/2021, 5:36 PM
    Hi, I am trying to gauge Prefect for our workflow management needs. I had a question regarding self-hosted Prefect server but couldn’t find supported documentation for this : Is there a way to scale the Prefect Server k8s deployment to more than 1 replica on increased load? An analogy to Airflow scheduler scaling introduced in 2.0 - https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html#running-more-than-one-scheduler
    k
    • 2
    • 3
  • a

    Arun Dass

    08/24/2021, 10:11 PM
    how can I access the prefect cloud graphql api using an api key? Do I somehow create a jwt token from the api key?
    k
    • 2
    • 5
  • r

    Ryan Smith

    08/25/2021, 7:00 AM
    Hi all, we're testing out running prefect server with an ECS Agent and Docker Containers for flow storage. Everything is working great so long as we use the LocalExecutor or LocalDaskExecutor for our flows. However, when we try to use the DaskExecutor to launch a temporary Fargate dask cluster, the worker nodes on that temporary dask cluster start complaining about not being authenticated with the cloud service. Any ideas about what is going wrong here? Has anyone else attempted this kind of setup before?
    k
    v
    m
    • 4
    • 22
  • b

    Bruno Murino

    08/25/2021, 9:01 AM
    Hi everyone — I have a flow that runs every 10 minutes and it’s been pretty consistent, however every now and then the flow throws an exception 502:
    502 Server Error: Bad Gateway for url: <http://prefect.landbay.internal:8081/api/graphql/graphql>
    however I’m not sure there’s an issue with how I’ve configured the URLs as the flow runs 99% of time without raising this….. For reference, these are the only 2 URLS I set:
    PREFECT__SERVER__ENDPOINT=<https://prefect.prodenv.landbay.co.uk/api/graphql>
    PREFECT__API__HOST=<https://prefect.prodenv.landbay.co.uk/api/graphql>
    n
    • 2
    • 1
  • p

    Pierre Monico

    08/25/2021, 9:04 AM
    I now have a working Prefect Server with flows running within a
    DockerRun
    run config. However, since I deployed to the server the runs are not very reliable. I keep getting random errors in a totally non-deterministic way (cf. image). The flows all run fine on my machine and were also running fine when executed in a Docker container I was managing myself (on cloud infrastructure). Some of the errors include: •
    HTTPConnectionPool(host='host.docker.internal', port=4200): Read timed out. (read timeout=15)
    •
    sqlalchemy.exc.NoSuchTableError
    (I am writing to Postgres tables - the table does exist and sometimes the run even succeeds) •
    TypeError: 'NoneType' object is not subscriptable
    •
    No heartbeat detected from the remote task; retrying the run.This will be retry 1 of 3.
    I believe most of the errors are related to some time out issue / the state of the task can’t be monitored but I am confused as to why this happens. My VM has sufficient resources (I checked the monitoring) but I am thinking it might be worth scaling it up? I know the question might be a bit broad but would be happy to know if anyone experienced something similar and / or knows the reasons / a fix.
    k
    • 2
    • 6
  • m

    Michael Hadorn

    08/25/2021, 12:42 PM
    We got this error also in the local dev machine version (on-prem):
    Expected type UUID!
    see here: https://github.com/PrefectHQ/prefect/issues/4687 Can you fix this also in the server edition?
    k
    • 2
    • 5
  • t

    Tom Forbes

    08/25/2021, 1:59 PM
    Is there a simple way to disable the
    Task 'xy[z]': Finished task run…
    messages?
    p
    k
    • 3
    • 7
  • l

    Lawrence Finn

    08/25/2021, 3:36 PM
    Is there a way to retrigger a failed task run via the graphql api?
    k
    • 2
    • 4
  • g

    Gaylord Cherencey

    08/25/2021, 9:41 PM
    I saw this on Prefect Core website:
    COMING SOON - Task Affinity Run each of a flow's tasks in a completely different environment, including new dependencies or platforms.
    How far this feature is and will this mean that we could define a Docker image per task for example?
    n
    • 2
    • 2
  • r

    Rachel

    08/25/2021, 10:16 PM
    [SOLVED]~Has anyone had trouble getting the server to start on a mac?~
    subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 15.
    and later
    subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 15.
    n
    • 2
    • 4
  • y

    YD

    08/25/2021, 10:51 PM
    Tasks execution when one of the task fails. if I have a situation:
    @task(timeout=60)
    def task_1():
        x = func1(...)
        return x == 0
    
    @task(timeout=60)
    def task_2(r):
        y = func2(...)
        return y == 0
    
    @task(timeout=60)
    def task_3():
        z = func3(...)
        return z == 0
    
    def main():
        with Flow("my-flow", executor=LocalDaskExecutor()) as flow:
            r = task_1()
            r = task_2(r)
    
            r2 = task_3()
    if
    task_1
    times out and fail,
    task_2
    will not run. What can I change so that I can still use the
    LocalDaskExecutor
    but allow
    task_2
    to run even if
    task_1
    failed ?
    n
    b
    • 3
    • 4
  • t

    Tom Tom

    08/25/2021, 11:46 PM
    Hey everyone, on my local agent everything works fine. But if i want to use 
    DockerRun(image="prefecthq/prefect")
      with
    prefect agent docker start
     i´m getting this Error:
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'C'")
    I know that i can´t import other dependencies but i only import following modules from prefect in the script:
    from prefect import Flow, task
    from prefect.run_configs import LocalRun,DockerRun
    Or is it possible to run 
    DockerRun()
     in an existing Container?
    n
    k
    • 3
    • 24
  • s

    Sumit Kumar Rai

    08/26/2021, 3:14 AM
    Hello everyone, is there a way to manually run a Prefect flow from the middle of the schematic in Prefect Server(Prefect Cloud)? Sometimes my flow fails and I need to skip tasks and run from the middle.
    k
    • 2
    • 3
  • p

    Pierre Monico

    08/26/2021, 8:40 AM
    When you register your flows from code (Python), using e.g.
    Module
    storage with
    stored_as_script=True
    , why do all the flows and their dependencies need to be resolved? e.g. all my flows import
    db
    which is an instance of
    sqlalchemy.create_engine()
    that I setup with an environment variable. If I try to register e.g. from a CI pipeline, I have to set all those variables just to be able to register a script that will be stored as a file anyway? (NB: can’t use the CLI
    prefect register
    because it is not able to resolve any of the dependencies of my flows (they are in a folder), spitting
    ModuleNotFoundError
    )
    m
    • 2
    • 6
  • r

    Rachel

    08/26/2021, 4:05 PM
    I deployed the server on a VPS and am trying to start a local agent, but it seems to expect an API key. When I run locally, this doesn’t happen. Is there a config I’m missing to tell it to run locally?
    k
    • 2
    • 2
  • h

    Henry

    08/27/2021, 2:10 AM
    Hello. I'm upgrading to 0.15.4 and it seems that we're receiving an error message from Alembic when graphql tries to run migrations
    Could not upgrade the database!
    Error: Can't load plugin: sqlalchemy.dialects:postgres
    k
    m
    • 3
    • 23
  • b

    Ben Collier

    08/27/2021, 10:31 AM
    Hi all - could someone point me to a document which goes over, in some detail, what happens to a workflow running on an agent if the agent is stopped in the middle of the workflow?
    k
    b
    • 3
    • 3
  • m

    Marcus Hughes

    08/27/2021, 3:42 PM
    I see in Dask that there's a priority parameter for "when there's more work than workers [and] Dask has to decide which tasks to prioritize over others." Is this exposed to Prefect? More specifically, let's say I have a collection of flows and my system is running low on resources, is there a way to tell Prefect to prioritize completing one flow over another? Thanks in advance!
    n
    k
    • 3
    • 2
  • w

    Wilson Bilkovich

    08/27/2021, 6:37 PM
    Sorry for mentioning this again, but I’m still grappling with this error. Does this look familiar to anyone?
    ERROR:asyncio:Fatal error on SSL transport
    k
    • 2
    • 16
  • s

    Serdar Tumgoren

    08/27/2021, 6:52 PM
    Hey folks, I’m trying to set up a workflow where we spin up a GKE cluster on demand, run a Kubernetes + Dask workflow on it, and then tear down the GKE cluster when the flow is finished. Is this the kind of thing that I’d want to do with a flow inside of a flow? In other words, have a parent flow that manages the cluster, while the child flow handles the data ETL work I’m performing on K8S+Dask? If so, are there any drawbacks to this approach I should be aware of (e.g. visibility into potential errors in the child flow)?
    k
    w
    e
    • 4
    • 45
  • m

    May

    08/27/2021, 8:19 PM
    Something went wrong while trying to fetch failed flow information. Please try refreshing your page. If this error persists, return to this page after a few minutes.
    latest prefect-server deployment via helm - We get this error after 2 weeks of flows runs - now we cantt see our flows 😞
    n
    • 2
    • 5
  • m

    Michael

    08/30/2021, 8:14 AM
    Hey folks, apologies if this is a rookie question, as I’m just getting started with Prefect. What I’m aiming to do is set up a flow where each task is a container run on a separate google cloud compute instance (where the instance is spun up on task start and destroyed on task end, instance type & config depends on the task). It seems like this lies somewhere between the purpose of the Docker agent and the Kubernetes agent, where the former runs task as containers but only locally, while the latter can create new instances but requires a full Kubernetes setup (which we currently don’t use). Am I understanding this correctly? Is the workflow I described similar to what someone has built with Prefect?
    👀 1
    w
    k
    • 3
    • 7
  • c

    Casey Green

    08/30/2021, 8:38 PM
    Is it possible to programmatically archive a flow? I found this but
    mutation { archive_flow(...) { ... } }
    doesn't appear to be valid anymore, I only see
    delete
    mutations.
    k
    • 2
    • 5
  • s

    Stanislav Molleker

    08/31/2021, 3:50 PM
    Hi everyone. Does anybody have some practice example of using Google Cloud Tasks? (Google and GitHub didn’t help me, unfortunately).
    k
    • 2
    • 3
  • m

    Michail Melonas

    09/01/2021, 2:25 PM
    I am having trouble passing sensitive information to my flow's run configuration. Having:
    secret = PrefectSecret("SOME_SECRET")
    flow.storage = S3(bucket="some-bucket", secrets=["AWS_CREDENTIALS"])
    
    flow.run_config = KubernetesRun(
        image="<http://registry.gitlab.com/<some_image>|registry.gitlab.com/<some_image>>",
        labels=["prefect-14"],
        env={
            "PREFECT__LOGGING__LEVEL": "DEBUG",
            "SECRET": secret
        },
        image_pull_secrets=["image-puller"],
    )
    and then running:
    flow.register("ProjectName")
    I get:
    TypeError: Object of type PrefectSecret is not JSON serializable
    Any thoughts on this?
    k
    c
    • 3
    • 21
  • j

    Jeevan Acharya

    09/01/2021, 2:56 PM
    Hi Everyone 👋 I have setup the prefect server in a kubernetes cluster using Helm Chart (https://github.com/PrefectHQ/server/tree/master/helm/prefect-server) I have also set up ingress rules for the UI and apollo components. Unfortunately the UI page is not loading, i am able to access apollo though. I have created the tenant as mentioned in the docs using '--set jobs.createTenant.enabled=true' and also directly using graphql mutation. Any idea what might be the issue?
    k
    m
    j
    • 4
    • 29
  • a

    Alexander van Eck

    09/02/2021, 9:55 AM
    Hello 👋 I would love to use Parameter and mypy together. However currently when I do;
    var: str = Parameter('var')
    mypy complains with
    Incompatible types in assignment (expression has type "Parameter", variable has type "str")
    Even though after calling this will be a string 🤔 Any thoughts on how to achieve this?
    b
    z
    • 3
    • 3
  • a

    ash

    09/02/2021, 10:05 AM
    Hello everyone , I am running my prefect on K8 cluster using helm charts. I have a separate postgres stateful database attached to persist states. I have 2 questions here :
    1.) I have flow which gets data from an api and i want to publish it to slack directly, how should i do it without writing it anywhere. 
    2.) If i want to save my flow/task results on a machine from where I am registering my flows despite flow running on pods on k8 ,how should i do it.
    z
    • 2
    • 5
  • a

    Anton Chekmarev

    09/02/2021, 12:06 PM
    Hello everyone, I am looking for a way to make my flow dependent on another one. In Airflow I used externalTaskSensor: in DAG "B" check DAG "A" status, if it's not a success then retry later. I've tried StartFlowRun, but it starts a new run of DAG "A" every time. I have some heavy computation in "A" and don't want it to be run multiple times. Is there a way to do it without kicking off parent flow?
    👀 1
    k
    • 2
    • 1
  • a

    Alfie

    09/03/2021, 6:25 AM
    Hi Team, I’m using Prefect server 0.13.10. Sometimes a scheduled task could exit unexpectedly, and seems Lazarus will re-schedule it after a while. In my case, I want to just let it go and do not want to have it re-scheduled. It’s said that I can be configured per flow in Prefect UI. But what I want is to set this when I create the flow. Is that possible? Thanks.
    k
    • 2
    • 13
Powered by Linen
Title
a

Alfie

09/03/2021, 6:25 AM
Hi Team, I’m using Prefect server 0.13.10. Sometimes a scheduled task could exit unexpectedly, and seems Lazarus will re-schedule it after a while. In my case, I want to just let it go and do not want to have it re-scheduled. It’s said that I can be configured per flow in Prefect UI. But what I want is to set this when I create the flow. Is that possible? Thanks.
k

Kevin Kho

09/03/2021, 1:45 PM
Hey @Alfie, see if you have this in your config.toml and then set it to 0
a

Alfie

09/03/2021, 3:50 PM
Thanks @Kevin Kho, so the env for this config should be
PREFECT___SERVICES____LAZARUS___RESURRECTION_ATTEMPT_LIMIT
, right?
k

Kevin Kho

09/03/2021, 3:54 PM
I think that’s right, but not 100% sure it’s available on your version (but I think this should be)
a

Alfie

09/03/2021, 3:54 PM
Let me try it now, thanks!
k

Kevin Kho

09/03/2021, 3:54 PM
You might have to restart server Actually that looks like a global setting, let me look for the flow setting
Ignore the above, that’s what I would try
a

Alfie

09/03/2021, 3:58 PM
Global setting is fine too
In Kubernetes, it’s the env for container prefect-scheduler, right? Or it’s for agent?
k

Kevin Kho

09/03/2021, 4:05 PM
Both should work I think
a

Alfie

09/03/2021, 4:31 PM
Thanks. I just set it on the agent, it looks good so far, will keep an eye on it.
Seems it does not work for agent, will try to set it on prefect-scheduler
[2021-09-03T16:05:02.478331+00:00] [info] [7] [140287568955136] [agent] msg="Deploying flow run 7b857363-db38-4a93-bac0-dcdce8139430"
the first time failed, and it’s retried after 10 min
[2021-09-03T16:15:10.798263+00:00] [info] [7] [140287568955136] [agent] msg="Deploying flow run 7b857363-db38-4a93-bac0-dcdce8139430"
k

Kevin Kho

09/03/2021, 5:06 PM
Ok if that still fails, I think you need to hit the
disable_flow_lazarus_process
graphql API endpoint
a

Alfie

09/06/2021, 6:32 AM
disable_flow_lazarus_process
works, thanks @Kevin Kho
👍 1
View count: 2