• justabill

    justabill

    1 year ago
    Hi everyone! Yesterday, we released Prefect 0.15.0 🚀. This release improves the Prefect Core Python API for running and inspecting flow runs, exposing some powerful new patterns. We've added: • A new CLI for running flows, prefect run, which makes the transition from local execution to running with a Cloud or Server backend seamless • The ability to run flows without an agent, which lets you run your flow locally with the rich set of features enabled by the backend API. • New tasks, create_flow_run and get_task_run_result, to create subflows and retrieve data from them. • New objects, FlowRunView, TaskRunView, FlowView, and TenantView, which enable you to pull the most recent information about flow and task runs from the backend without writing GraphQL queries • A new suite of CLI commands for managing authentication with API keys, simplifying authentication while retaining backwards compatibility with tokens. For more information and examples of how to use these features, check out this blog post. For the full list of enhancements and features, see the release changelog. We're excited to see what you can do with them. If you have a question or feedback about this release, we want to hear it! As always, you'll find us #prefect-community channel. Special thanks to @Stéphan Taljaard and @Tenzin Choedak for contributing to this release!
  • Jeremiah

    Jeremiah

    1 year ago
    This morning, Ben Lorica released a podcast with our very own @Chris White covering a lot of the motivation behind Prefect and starting to touch on the near-term roadmap (more to come on that very soon!). In addition, Ben and his colleagues are conducting a survey on the state of data engineering. Let’s make sure Prefect is well-represented! https://www.surveymonkey.com/r/DEStwitter
  • justabill

    justabill

    1 year ago
    Hey there! We've been happy to see the enthusiasm for the major improvements made in the 0.15.0 release. With the help of some community contributors, we've kept the momentum going with the most recent patch releases, including this week's 0.15.3 release. Cumulatively, these releases include: • Enabled retrieval of secrets from private Vault instances with new
    VaultSecret
    task • Expanded configuration to expose deeper settings for Docker flow runs • Enabled loading of additional repository files with
    Git
    storage • Added documentation for querying account role based access control data • Added
    --no-schedule
    flag to
    prefect register
    to disable flow schedules when registered • Enabled recursive flow registration and builds through globs • Enabled calculation of evaluation parameters on the fly through new
    evaluation_parameters
    parameter for the
    RunGreatExpectationsValidation
    task • Increased robustness of flow runs to connection errors through improved exception handling Special thanks to @Gabriel Montañola, @Nelson Griffiths, Pawel Janowski, @Yueh-Han, and @Chris Ottinger for contributing to these enhancements!
  • justabill

    justabill

    1 year ago
    We've been working on some exciting 😛refect: updates that we're looking forward to announcing 🔜. In the meantime, our team and community members have squashed 🐛 and made incremental enhancements in last week's 0.15.5 Release. Here are some highlights: • Python 3.9 support is stabilizing and we are now publishing Docker images e.g.
    0.15.5-python3.9
    . • While registering very large flows, it was possible to see timeouts or incomplete registration. Now, flow edges and tasks are registered in batches allowing for much larger flows. • Agentless flow runs now receive the environment variables of the process that starts them; this makes agentless runs easier to use in production. •
    --idempotency-key
    support has been added to
    prefect run
    prefect run
    will now ignore schedules on local flow runs by default and only a single run will be performed. A
    --schedule
    flag was added if you’d like the command to go forever and run the flow on a schedule. • Several task library improvements have been made, including a new
    SnowflakeQueryFromFile
    task, improved logging in the AWS
    BatchSubmit
    task, improved runtime handling for MySQL Tasks, and updated authentication for the
    DremioFetch
    task. • Log limits have been updated to reduce errors when many large logs are sent.Note that this release includes several breaking changes: • In a secure-by-default effort, services run by the
    prefect server
    cli now listen to localhost by default and are not accessible from other machines; use
    --expose
    to allow connections from a remote location. • Prefect Server will need to be upgraded to version
    2021.09.02
    before flows can be registered from this version of Core. Special thanks to Deepyaman Datta, Henri Hannetel, Johnny Tirado, @Kathryn Klarich, @Tenzin Choedak, and Vincent Xue for contributing to this release :thank-you:!
  • Chris Reuter

    Chris Reuter

    11 months ago
    Hi everybody 👋! We're going to have a huge week next week. If you want to see the next generation of Prefect tech in action, sign up for our livestream with @Laura Lorenz on October 8th at 4p Eastern/1p Pacific. Sign up on Meetup, or attend the Livestream directly on

    Youtube

    . See you there!
  • Chris White

    Chris White

    11 months ago
    Hello everyone <!here>! I'm extremely excited to finally share what we've been working on the last few months: Prefect Orion, our second generation workflow orchestration tool. If you've ever asked about dynamic tasks, subflows, self-hosting Server without Docker, or almost any other common question over the last few years, the chances are you'll find the solution with Orion.  So much of what we sought to achieve was driven by this community's questions and feedback, and I can't emphasize enough how great it's been to collaborate with you all! Now that this is live, please peruse the docs, test out the initial alpha release and give us feedback!  We want you all to be as excited as we are for the possibilities here and help us shape Orion's near term roadmap. To get started: • read about the vision of Orion directly from @Jeremiah: https://www.prefect.io/blog/announcing-prefect-orion/ • begin building and testing your new workflows with our revamped documentation: https://orion-docs.prefect.io/
  • Kevin Kho

    Kevin Kho

    11 months ago
    Hi everyone, hot on the heels of our Orion announcement, we wanted to get feedback from the community on how you’re using Prefect today, where you’re finding value, and how we can improve. It should take 3-5 minutes. And you’ll be entered in a drawing to win Prefect swag! https://prefect.qualtrics.com/jfe/form/SV_41MC3mkByz18Ok6
  • Chris Reuter

    Chris Reuter

    11 months ago
    Hi everyone 👋! Hot on the heels of the Orion announcement, we have some great livestreams coming up that will feature Orion: -Today at 4p Eastern/1p Pacific @Laura Lorenz will host another spooky October livestream and take Orion for a spin. She'll highlight the new API, subflows and caching, as well as the killer new UI and deployment details. Sign up here to attend! -On Friday, October 22nd at 4p Eastern/1p Pacific Prefect CEO @Jeremiah and CTO @Chris White will host a Fireside Chat focused on Orion. Anyone in the community is invited to join and ask questions about whatever is on your mind. Sign up here to attend!
  • Chris Reuter

    Chris Reuter

    11 months ago
    Hi all 👋 - more Prefect content coming in 🔥! Scaling Prefect with Dask is always a hot topic in the community. Our friends at Coiled are hosting a livestream with Prefect CTO @Chris White and Coiled CEO @Matthew Rocklin, covering how to scale everything from a single process to an entire auto-scaling cluster. Join them on 10/26 at 6p Eastern/3p Pacific by registering here.
  • Chris Reuter

    Chris Reuter

    11 months ago
    Hi there Prefect community :marvin-duck:! Coming up in 2 short days on Oct 22 at 4p Eastern we'll be hosting a 🔥 Fireside Chat with @Jeremiah and @Chris White. If you haven't signed up yet you can do so here. The primary focus of this livestream will be Orion🌠, but any and all questions are welcome. If you'd like to submit a question ahead of time you can do so here. If you have any questions live, you can also use the Youtube chat to ask them - we'll be monitoring it throughout the stream. I can't wait to see you all there! Until then, happy Prefecting.
    Chris Reuter
    1 replies
    Copy to Clipboard