https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • a

    Adam Brusselback

    03/09/2021, 4:35 PM
    If there was work done to implement tenants for server, would a PR be entertained? Or is that a hard segregation you are trying to maintain at all costs between cloud / server?
    j
    g
    • 3
    • 5
  • c

    Charles Leung

    03/10/2021, 6:45 PM
    Hello Prefect Team, I'm trying to use local secrets with the core server setup to test some things. However, I constantly get an key error even as i set the environment variable via the agent as such: prefect agent start docker --env PREFECT__CONTEXT__SECRETS__VAULT_TOKEN=<REDACT> --api <REDACT> Unexpected error: HTTPError('400 Client Error: Bad Request for url: <REDACT>/graphql') Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/prefect/client/secrets.py", line 137, in get value = secrets[self.name] File "/usr/local/lib/python3.7/site-packages/box/box.py", line 335, in getitem raise BoxKeyError(str(err)) from None box.exceptions.BoxKeyError: "'vault_token'" Am i missing something in my setup? use_local_secrets is set to true in the config.
    j
    • 2
    • 14
  • j

    Josh Greenhalgh

    03/11/2021, 4:43 PM
    Hi wonder of anyone could help? I have a colleague running windows and they get the following error when they run
    prefect server start
    - traceback in thread
    m
    • 2
    • 7
  • a

    ale

    03/11/2021, 6:57 PM
    Hey folks 🙂 I've started experimenting with ECS agent to replace our current Fargate agent, but I'm having issues with AWS permissions. Is there a place where I can find the minimum set of permissions that an ECS agent needs to successfully deploy a flow as an ECS task?
    m
    m
    +3
    • 6
    • 31
  • c

    Chohang Ng

    03/11/2021, 8:20 PM
    Hi all, I've built an ETL pipeline with prefect locally and know that I need to run the prefect local agent start to pick up scheduled workflows. I am currently doing it on Powershell. My question is if I close the powershell, how do I know if the agent is still running? Is there a way to make sure it is always running so that my scheduled tasks will run on schedule?
    a
    • 2
    • 3
  • c

    Chohang Ng

    03/11/2021, 9:55 PM
    I am trying to deploy the prefect pipeline to the window server 2012 r2 but it doesn't work since docker is not compatible with r2. Is there any other workaround or do I have to use linux to run prefect?
  • l

    Levi Leal

    03/12/2021, 2:29 PM
    Trying to add DaskExecutor to a flow I run on K8s. I'm currently using KubernetesRun and Docker for storage. When I add DaskExecutor I get and error while building the flow image. On the thread the code I'm using
    • 1
    • 5
  • j

    Josh Greenhalgh

    03/12/2021, 2:55 PM
    Hey all - I am registering my flows with the following (to a server deployment on k8s);
    def register_deploy():
        # pylint: disable=missing-function-docstring
        with prefect.context(
            config={
                "cloud": {"graphql": os.environ["GRPAHQL_ENDPOINT"]},
                "server": {"ui": {"endpoint": os.environ["UI_ENDPOINT"]}},
            }
        ):
    
            for flow_name, flow in all_flows.items():
                <http://logging.info|logging.info>(f"Registering flow: {flow_name}")
                flow.register(project_name=PROD_PROJECT, labels=[SHARED_LABEL])
    This works (in terms of registering) but the stdout is the following;
    Result check: OK
    Flow URL: <http://localhost:8080/default/flow/a6013647-a31e-4d3c-9e7e-82e69c02f71e>
     └── ID: d78a0fc5-3990-461d-91d7-593a4278b6b2
     └── Project: prod
     └── Labels: ['prefect-agent']
    I thought setting
    config.server.ui.endpoint
    would lead to that endpoint being printed but it seems to not (
    localhost:8080
    as opposed to
    UI_ENDPOINT
    ) - any ideas?
    m
    • 2
    • 12
  • j

    Josh Greenhalgh

    03/12/2021, 6:16 PM
    Hi Again; I am running into the following error when registering flows on CI (works perfectly from my local machine);
    n
    • 2
    • 13
  • j

    Josh Greenhalgh

    03/15/2021, 3:28 PM
    Haha Me again! So I am having some trouble with attaching
    Result
    classes to my flows. If I specify the result class in the same file that the flow is defined via
    flow.result = ...
    everything works as expected. However if I import the flow def and then attach the result class (same way just outside the flow def) I do not get any stored results. I have put together a minimal repo here -> https://github.com/josh-gree2/minimal-repo/compare/result-in-cli...master (the two branches show what I mean)
    m
    m
    • 3
    • 21
  • g

    Gustavo Silva

    03/15/2021, 3:52 PM
    Hi everyone
  • g

    Gustavo Silva

    03/15/2021, 3:53 PM
    I would like to ask how do I get more information to find the issue with a failed flow. In my logs I juts got
    Process PID 266 returned non-zero exit code
    k
    m
    • 3
    • 6
  • d

    Dana Merrick

    03/15/2021, 5:22 PM
    is there a way to edit the command on k8s jobs? it's
    prefect execute flow-run
    but I want to add a command before that
    k
    • 2
    • 12
  • c

    Chohang Ng

    03/16/2021, 1:00 AM
    Could anyone help me understand updating flows? Do I always need to re-register all my flows and flow of flows? I built a script that register all my flows and flow of flow. However everytime I make an update to my flows, it takes a long time as if it is running a whole cycle on the backend. Am I doing it right?
    m
    • 2
    • 2
  • c

    Charles Leung

    03/16/2021, 3:02 PM
    Hey Team! i have some general questions about secrets and script storage. When i use Gitlab Storage to register flows, it seems that they still look for secrets i plan to add onto the agent during execution; yet an error is thrown when i'm just registering the flow:
    ValueError: Local Secret "VAULT_TOKEN" was not found.
    Is this the expected behavior? should all secrets be registered wherever flows are created/registered?
    m
    • 2
    • 30
  • j

    Josh Greenhalgh

    03/16/2021, 7:01 PM
    Hello again! So I have run into the following issue - seems to have appeared out of nowhere since don't think it was happening yesterday... I am registering my flows on CI - get the following output;
    INFO:root:Registering flow: weather_data_etl
    Result check: OK
    Flow URL: <http://localhost:8080/default/flow/b366cae3-8200-42d9-87b5-f621711e897a>
     └── ID: 2570985c-efd0-4b29-88f1-96b86862d34a
     └── Project: prod
     └── Labels: ['prefect-agent']
    INFO:root:Registering flow: test_flow
    Result check: OK
    Flow URL: <http://localhost:8080/default/flow/96b48708-c731-4dd7-9ba0-f354b8ef7028>
     └── ID: 1307c478-cc0e-49ee-b4b8-b00a8834eed6
     └── Project: prod
     └── Labels: ['prefect-agent']
    So all seems good... However if I go to the UI there are no flows visible at all - that being said if I look into the postgres DB then the flows with the above IDs both exist...
    n
    • 2
    • 5
  • j

    Josh Greenhalgh

    03/16/2021, 11:10 PM
    Could anybody give me some idea of sensible default resource limits for the helm chart? They are all unspecified. My apollo server seems to be having trouble with connections and I wonder if its because the deployment is not requesting enough resources? I have a few mapped tasks feeding into each other and all goes well until towards the end when some of the mapped tasks finish in a pending state - this is completely random behaviour some times everything works as expected. The particular error I get in the logs is this in the thread
    n
    a
    m
    • 4
    • 46
  • a

    Adam Lewis

    03/17/2021, 1:52 PM
    Hi everyone, I have prefect-server set up via minikube and the official helm charts (prefect-server-2021.03.10) including a Kubernetes Agent set up via helm chart. I want a flow to use the DaskExecutor so I've set the flow.executor to an instance of the DaskExecutor. I'm able to register and run the flow, but when I run the flow a job pod starts, but no other pods do where I would expect dask workers to spin up. Does anyone know what is going on here? The simple task and dask worker definitions can be seen at https://github.com/Adam-D-Lewis/sample-flow if needed.
    j
    • 2
    • 4
  • c

    Charles Liu

    03/17/2021, 11:20 PM
    Quick question: is there a setting or a flag that will not allow prefect to queue a flow that is identical to one that is currently running in the schedule queue?
    c
    • 2
    • 5
  • c

    Charles Leung

    03/18/2021, 5:00 PM
    Hey guys! is there a way to execute a registered flow via the prefect CLI? not using flow.run() but actually doing a "quick run" for example on the UI
    j
    • 2
    • 2
  • c

    Charles Liu

    03/18/2021, 10:56 PM
    Hi all! I'm looking to spin up a second K8 agent in the same cluster as a dev agent, but it does not seem to appear when following the kubectl install options. Any pointers?
    c
    • 2
    • 8
  • m

    Michael Stolte

    03/19/2021, 12:04 AM
    Hello, I'm having an issue starting prefect server with persistent storage on Windows 10, any help would be great! I've tried using my own config.toml file where the only different is the UI's Apollo URL. I've also tried a docker-compose file ( however I admit I'm not the best with docker )
    n
    • 2
    • 9
  • c

    Christian

    03/19/2021, 11:23 AM
    Struggling after Update to 0.14.12 to start prefect-server. Hi all 👋 . I just upgraded to prefect 0.14.12 (from 0.14.4) running server on a machine of us (hosting more docker apps, so maybe something changed outside of my control as other people use it to start apps, too). I do connect to the system via VPN which might create more complications? After the update my command:
    prefect server start --use-volume -d
    is pulling the images, creating the network and creating the parts (tmp_postgres_1, tmp_hasura_1, …). However, it never finishes (no detach) and I do get errors in the logs of tmp_apollo_1:
    Trying again in 3 seconds...
    Building schema...
    FetchError: request to <http://hasura:3001/v1alpha1/graphql> failed, reason: connect ECONNREFUSED 192.168.64.3:3001
        at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1461:11)
        at ClientRequest.emit (events.js:315:20)
        at Socket.socketErrorListener (_http_client.js:469:9)
        at Socket.emit (events.js:315:20)
        at emitErrorNT (internal/streams/destroy.js:106:8)
        at emitErrorCloseNT (internal/streams/destroy.js:74:3)
        at processTicksAndRejections (internal/process/task_queues.js:80:21) {
      type: 'system',
      errno: 'ECONNREFUSED',
      code: 'ECONNREFUSED'
    }
    Trying again in 3 seconds...
    Not sure what’s going on? My config is pretty simple, too:
    [server]
        [server.database]
        host = "localhost"
        port = "5432"
        host_port = "5434"
    
        [server.hasura]
        host = "localhost"
        port = "3000"
        host_port = "3001"
    The micro-services seem to be up and running:
    (prefect) [root@myserver:~]$ docker ps
    CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS                    PORTS                              NAMES
    aec1a48767a3        prefecthq/ui:core-0.14.12       "/docker-entrypoint.…"   17 minutes ago      Up 17 minutes (healthy)   80/tcp, 0.0.0.0:8080->8080/tcp     tmp_ui_1
    2996c1840fe1        prefecthq/apollo:core-0.14.12   "tini -g -- bash -c …"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:4200->4200/tcp             tmp_apollo_1
    2263ca7701e6        prefecthq/server:core-0.14.12   "tini -g -- python s…"   17 minutes ago      Up 17 minutes                                                tmp_towel_1
    ed29fdb1b3b0        prefecthq/server:core-0.14.12   "tini -g -- bash -c …"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:4201->4201/tcp             tmp_graphql_1
    2a3dffb7d641        hasura/graphql-engine:v1.3.3    "graphql-engine serve"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:3001->3000/tcp             tmp_hasura_1
    cafbde90dc48        postgres:11                     "docker-entrypoint.s…"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:5434->5432/tcp             tmp_postgres_1
    n
    • 2
    • 11
  • c

    Christian

    03/19/2021, 11:23 AM
    Any help would be much appreciated!
  • m

    Massoud Mazar

    03/19/2021, 3:08 PM
    Hi everyone, I upgraded my Prefect server from
    0.14.6
    to
    0.14.12
    and since then I see even when idle, CPU hovers between 10%-50%.
    docker stats --all --format "table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"
    shows the following:
    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT
    c6eda45c6f59        tmp_ui_1            0.00%               4.953MiB / 7.432GiB
    2466edd82357        tmp_towel_1         0.00%               52.57MiB / 7.432GiB
    e9d0643d134e        tmp_apollo_1        2.69%               62.68MiB / 7.432GiB
    ad41dcf8646a        tmp_graphql_1       3.24%               69.5MiB / 7.432GiB
    a8f65bcb2efa        tmp_hasura_1        4.62%               153.8MiB / 7.432GiB
    bc9742482bd8        tmp_postgres_1      4.64%               27.86MiB / 7.432GiB
    n
    • 2
    • 4
  • d

    Dana Merrick

    03/19/2021, 3:38 PM
    I accidentally clicked "Cloud" instead of "Server" when configuring my server's graphql endpoint... now I get the following error when I try to visit my server:

    https://i.imgur.com/RhqSS8A.png▾

    j
    n
    • 3
    • 5
  • d

    Dana Merrick

    03/19/2021, 3:39 PM
    it works in an incogneto tab... is there a way to correct this without wiping all of my browser storage?
    m
    • 2
    • 2
  • p

    Paul Prescod

    03/21/2021, 3:30 PM
    Hello everyone! I was thinking about a situation where we might use Prefect and the following unusual? unique? challenge came to mind. Our tasks/flows can include custom code written by customers. It's messy, but a very powerful model which allows them to write specialized tasks. What we would want is for each worker container to have work from only one customer. The scheduler could spin up multiple container per customer, but never multiple customers' work in a single container. Is there any way to implement that in Prefect without falling back to a single container per flow?
    d
    • 2
    • 5
  • l

    Lukas N.

    03/23/2021, 9:42 AM
    Hello everyone 👋. We're hosting prefect server in k8s and also running our flows there using the k8s agent. Everything works great, we have a Slack notification (state handler) setup to notify us when something goes wrong. However, when the pod running the flow is killed externally (e.g. due to OOM), we get no slack notification. This makes sense because the notifications happens from within the flow pod. Is there a way how to send these kind of notifications from other Prefect server components (towel) ? Or should this case be handled on kubernetes cluster with OOMkill alerting?
    m
    • 2
    • 2
  • d

    Dana Merrick

    03/23/2021, 4:15 PM
    what's the purpose of
    RunNamespacedJob
    ? do I add it inside a task, so a task can create a Job during the course of a Flow?
    m
    • 2
    • 3
Powered by Linen
Title
d

Dana Merrick

03/23/2021, 4:15 PM
what's the purpose of
RunNamespacedJob
? do I add it inside a task, so a task can create a Job during the course of a Flow?
m

Mariia Kerimova

03/23/2021, 6:37 PM
Hello Dana! You can use
RunNamespacedJob
task to run a namespaced job. No need to add it inside the task. You can initialize your
RunNamespacedJob
instance, pass namespace, job template, etc. Here is a small example
from prefect.tasks.kubernetes.job import RunNamespacedJob
from prefect import Flow, task

body = {
    "apiVersion": "batch/v1",
    "kind": "Job",
    "metadata": {
        "name": "pi"
    },
    "spec": {
        "template": {
            "spec": {
                "containers": [
                    {
                        "name": "pi",
                        "image": "perl",
                        "command": [
                            "perl",
                            "-Mbignum=bpi",
                            "-wle",
                            "print bpi(2000)"
                        ]
                    }
                ],
                "restartPolicy": "Never"
            }
        },
        "backoffLimit": 4
    }
}

your_task_name = RunNamespacedJob(body=body, namespace="test")


with Flow("Test") as flow:
    your_task_name(kubernetes_api_key_secret=None)

if __name__ == "__main__":
    flow.run()
d

Dana Merrick

03/23/2021, 7:26 PM
I see, so it's like a helper method that can be used for general purposes
it uses the underlying Prefect k8s library to instantiate a k8s Job instead of using the k8s client API directly
💯 1
View count: 2