• psimakis

    psimakis

    1 year ago
    Hello everyone! Using tasks with manual approval is a nice way to semi-automate the flows that could not be fully automated yet. A great feature about paused tasks could be to support a time-out functionality. When a time-out value is passed, paused task will stay paused only for a period of time and the user will always has the ability to resume/approve the task earlier that the time-out. A use case: Use prefect to start an ECS instance for N hours but allow user to stop the instance any time he thinks that the task in ECS instance is done. Always stop instance after N hours to prevent extra charges. I know that this feature is currently not supported but I would like to know if you find it is useful and also how this functionality could be implemented with prefect.
    psimakis
    1 replies
    Copy to Clipboard
  • Marion

    Marion

    1 year ago
    Hi everyone nice to meet you all. I have to run Prefect's flows from my API, I was wondering if you had a REST endpoint to run a flow, instead of using the prefect pylib.
    Marion
    Jenny
    8 replies
    Copy to Clipboard
  • Severin Ryberg [sevberg]

    Severin Ryberg [sevberg]

    1 year ago
    Hi all! We've recently run into the issue that very short tasks which should run in less than a second (tested when using a LocalAgent) can take much longer than that when using a KubernetesAgent. Unsurprisingly, it appears the culprit is the overhead time it takes for the DaskKubernetesEnvironment to spin up and tear down a worker pod, which is done for each task invocation. • Is it possible to avoid this overhead (or at least only pay it once), by telling the KubernetesAgent and/or DaskKubernetesEnvironment to always keep X worker pods alive which continue to wait for and complete tasks without dying until the end of the flow? • Or, if the above isn't possible, then perhaps the only thing to do here is have less finely-resolved tasks so that the overhead is not so noticeable. In such a case, what is the suggested single-task execution time when using a KubernetesAgent / DaskKubernetesEnvironment? • Possibility for other routes?
    Severin Ryberg [sevberg]
    Jim Crist-Harif
    +1
    14 replies
    Copy to Clipboard
  • Marc Lipoff

    Marc Lipoff

    1 year ago
    Using the functional api, how can i set a dependency when the outputs of a task dont depend on another. for example, i have 3 tasks: read, delete write. I want to read some data, then delete the old stuff, and then right the new stuff. "deleting" does not take any inputs; but we want to wait until "reading" is done before we do it.
    Marc Lipoff
    emre
    12 replies
    Copy to Clipboard
  • b

    Ben Wolz

    1 year ago
    Hi all, I'm currently running a simple flow with a Dask Executor that's spinning up a FargateCluster, but I'm getting a strange error that I can't find anywhere else online. When I run my flow with a DaskExecutor set up for a FargateCluster, I get a set of RunTimeErrors which say "IOLoop is closed". I've attached a .txt file of the code as well as the stack trace, if anyone has any advice it would be greatly appreciated! Thank you in advance
    b
    Jim Crist-Harif
    5 replies
    Copy to Clipboard
  • Jeff Brainerd

    Jeff Brainerd

    1 year ago
    Hey 😛refect: team, a question about migrating to RunConfigs: We’re on Prefect 0.14.4 and looking to migrate off of LocalEnvironment (also using DaskExecutor + FargateAgent). We are currently relying on LocalEnvironment’s
    on_start
    arg in order to bootstrap django when our flow runs. I’m not seeing any replacement for that in the new RunConfig world. Did I miss something… or is there another pattern for this? Thanks!
    Jeff Brainerd
    Amanda Wee
    +1
    4 replies
    Copy to Clipboard
  • Amanda Wee

    Amanda Wee

    1 year ago
    Hi friends, I have flows using S3 storage with the local executor and local agent, all wrapped up in a Docker image that is deployed in an ECS service (as per my wider team requirements). It works, but I'm not satisfied with the fact that we're basically reserving a fair bit of CPU and memory on the EC2 instance(s) for a bunch of flows that collectively run across only two hours each day. One option might be to switch to Docker storage with the ECS agent to run the flows as Fargate tasks. The catch is that because of our ECS setup, this may mean some kind of Docker-in-Docker solution, which seems to be a mess. Another option might be to stick with S3 storage, but make use of the Dask executor (and
    dask_cloudprovider.aws.FargateCluster
    plus adaptive scaling) with the local agent. My thinking is that I could separately build a Docker image that has all the required flow dependencies (they're the same for all the flows) and upload that to ECR during CI/CD, and then the local agent can pull from ECR and initialise the Dask executor with this image, while also initialising it with the flows retrieved from S3 storage. Is this feasible?
    Amanda Wee
    Jim Crist-Harif
    +1
    21 replies
    Copy to Clipboard
  • Henry

    Henry

    1 year ago
    can a prefect flow's results be piped to another prefect flow using flow-to-flow?
    Henry
    1 replies
    Copy to Clipboard
  • Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Prefect 0.14.5 has now been released!!! 🎉 This release: • Improves the quality of logs, warnings, and errors across several parts of the codebase, leading to a better user experience • Adds support for installing extra dependencies at runtime in our
    prefecthq/prefect
    images via setting a
    EXTRA_PIP_PACKAGES
    environment variable • Includes several improvements to agents, notably including a new
    execution_role_arn
    kwarg to both ECSAgent/ECSRun run config Big thanks to our community contributors this week (Loïc Macherel and Thomas Baldwin) 👏 The full release notes can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.5. Happy Engineering!
  • Billy McMonagle

    Billy McMonagle

    1 year ago
    I have a question regarding
    KubernetesRun
    ... is there a direct way to set metadata labels on the flow run job/pod, or is a custom job template necessary? I'm specifically interested in tagging my jobs so that datadog is able to collect metrics.
    Billy McMonagle
    Jim Crist-Harif
    +1
    5 replies
    Copy to Clipboard