• h

    Howard Cornwell

    2 years ago
    Has anyone experienced using prefect for lots (millions) of API requests? I initially created one mapped task for each request, but this caused issues with prefect server performance. I’ve batched the requests into ~5000 per child task, but I can only retry entire batches, not single requests. I’m on the verge of tracking the state of each request outside of Prefect, but wonder if anyone has experienced / overcome similar issues?
    h
    Dylan
    4 replies
    Copy to Clipboard
  • e

    Eamon Keane

    2 years ago
    I'm still a little unclear on the kubernetes functionality in prefect (after reading the docs). Can I launch a
    CreateNamespacedPod
    task and will the prefect agent monitor the launched pod's exit status (via kubernetes api) as an indicator of task success/failure?
    e
    Dylan
    10 replies
    Copy to Clipboard
  • Darragh

    Darragh

    2 years ago
    Monitoring/Logging question - we’ve been seeing some odd behaviour that we’re still trying to classify, around parallell mapped tasks. We’ve rolled our own subflow semantics by taking a set of inputs [generally partitioned data files], and for numInputs we map over them and trigger a Flow with that input/file. This allows us to split out the work so we can have a bunch of Fargate nodes attack chunks of the data processing. Once these flows have been triggered, we have another mapped task that polls the flows for completion. By and large it works fine, but we see strange cases where some of the mapped tasks can do the following: • Submit and not start, so polling sits there forever waiting on it, or just fail because it never starts • Some mapped tasks, the pollers in particular, don’t actually get started for a few minutes after the others in the mapped input. For example -> mapped_lows = [flow_id_1, flow_id_2, flow_id_3] => poll_flow(mapped_flows) => poller_1, poller_2 start straightaway, poller_3 not triggered for several minutes with no indication as to why
    Darragh
    Laura Lorenz (she/her)
    8 replies
    Copy to Clipboard
  • j

    josh

    2 years ago
    Hey team, Prefect core version 
    0.12.1
     has been released! Here are a few notable highlights:📁   File-based storage for flows that unlocks the ability for flows to “hot reload”🚢   Flows stored in blob/bucket storage can be orchestrated with containerized agents/environments :dask:   Improved/streamlined environment and executor relationship🏗️   Task slugs are now stable across rebuilds of the same flow Read the full changelog here: https://github.com/PrefectHQ/prefect/releases/tag/0.12.1 Special thanks to all of the contributors who have assisted in discussions, issues, and PRs that have contributed to this release!
  • Jeff Brainerd

    Jeff Brainerd

    2 years ago
    Hi team, a Prefect mystery (at least for me 😬) — I’m using Prefect Cloud and Prefect Core 0.11.4, also Fargate Agent and docker storage. My goal is to call the Prefect API from a state handler, so I can report flow stats to Slack like flow duration when a flow finishes. Problem is I am getting an unauthenticated error. I can see from
    agent.py
    that the container env should include the env var:
    "PREFECT__CLOUD__AUTH_TOKEN": config.cloud.agent.auth_token,
    So at this point I’m not sure if that value is not set, or it is set correctly but the agent token doesn’t have the correct auth. PS — I am open to doing this another way, such as a dedicated SlackTask, but the state handler seems somehow more semantically correct, and the CloudHook doesn’t seem to provide that kind of detailed info. (Sorry for the long post.) Thanks! 🙏
    Jeff Brainerd
    Chris White
    +1
    10 replies
    Copy to Clipboard
  • h

    Hannah Amundson

    2 years ago
    Hello Prefect Community! I have a question about the KubernetesJobEnvironment. Is there a way to see what job environment has been stored/is being used? I would be fine with either something through the interactive API or getting into the docker image
    h
    Laura Lorenz (she/her)
    2 replies
    Copy to Clipboard
  • Jeremiah

    Jeremiah

    2 years ago
    👋 Hi everyone <!here>! A little less than a year ago, we created this Slack community just in case anyone would actually want to chat with us. The one commitment we made was to make this a welcoming place that would quickly result in an expert answer to any question. At some point earlier today, we welcomed the thousandth member of this community! That’s honestly mind-blowing to me, and from our entire team please know we’re very grateful for the interest and engagement of all of you. Over the last year, it’s been our pleasure to build relationships, develop features, discuss the future, and — just maybe once or twice — track down bugs with you. Thanks so much for being part of our community — I promise we’ve got plenty in store that you’re going to love. Happy engineering!
  • Alfie

    Alfie

    2 years ago
    Hi Team,
  • Alfie

    Alfie

    2 years ago
    I’m new to Prefect, and want to know if Prefect exposes any REST API. I have the frontend UI already and want to figure out a way to talk with Prefect
    Alfie
    Chris White
    2 replies
    Copy to Clipboard
  • Alfie

    Alfie

    2 years ago
    Thanks.