https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
announcements
  • k

    Kingsley Blatter

    01/07/2021, 8:09 PM
    💡Spotlight on several new features💡 It's been a while since I posted here, so lots to catch up on. Over the last couple of months, we've introduced several new features both in our UI, api and in our core engine. Take a look at our changelog to see some of the cool features the team has been working on! Wishing you all a Happy New Year.
    Spotlight Jan 2021.mp4
    :upvote: 9
    :marvin: 4
    🚀 11
    👍 1
  • h

    Hiro Rodriguez

    01/20/2021, 11:05 PM
    Prefect turns 3 tomorrow - come celebrate with us! Please join our team and community for a virtual happy hour tomorrow at 8pm ET / 5pm PT. See you there! https://spatial.chat/s/prefect-birthday-party
    🍾 10
    🤘 3
    🍻 8
    :prefect: 8
    :marvin: 9
    3️⃣ 14
    🎉 11
  • k

    Kingsley Blatter

    01/22/2021, 8:35 PM
    💡Spotlight on a new feature💡 Thank you to @Peyton Runyan for adding SQL Server to our task library. We now have three major relational database management systems included in our list of pre defined tasks. For anyone else that has an idea for a new task library and would like to contribute, here is a really great blog post by our very own @josh.
    :marvin: 10
    ❤️ 6
    🚀 10
  • j

    Jeremiah

    01/25/2021, 9:58 PM
    👋 Do you work for a 🚀 startup, 🎓 academic institution, 🌎 or non-profit? We’re running Lighthouse Partner programs for these three groups that include free access to Prefect Cloud. If you’d like to apply, please email hello@prefect.io. Thanks!
    :marvin: 7
    🚀 14
    🏠 5
    🎉 22
    💡 7
  • j

    Jeremiah

    02/09/2021, 3:38 PM
    🚀 :marvin-duck: :marvin: We have Liftoff! Today we’re announcing Prefect’s Liftoff initiative - a huge series of announcements that will greatly advance our product. All of this is thanks to the incredible growth and enthusiasm of this community. Though we confess that sometimes it’s a challenge for our small team to keep up with all of you (😅) we are committed to giving everyone here a first-class and positive experience and a large part of Liftoff is aimed at ensuring we are able to do so. Here are a few of the items we’re previewing today: • Prefect’s Series A financing • Our product vision for a modern workflow API • Success-based pricing for Prefect Cloud • Partnerships with Microsoft, Slate, and Coiled • Three new community programs: Prefect for Academia, Prefect for Startups, and Prefect for Nonprofits One very important thing that we’ll announce soon: a new referral program that will let us give back to this community to properly thank our product ambassadors. As always, we appreciate your excitement and are grateful for your interest. Stay tuned in the coming weeks for full details on each of our Liftoff announcements. Happy engineering!
    💯 6
    👍 2
    :marvin: 8
    :upvote: 3
    👽 3
    :cool-llama: 1
    ❤️ 3
    🚀 31
    🎉 4
    🙏 8
  • j

    Jim Crist-Harif

    02/10/2021, 4:57 PM
    Prefect 0.14.7 has been released 🚀 Highlights!: • Expanded support for docker networks within the Docker agent • Simplified the use of Secrets with `GitHub`/`Bitbucket`/`GitLab` storage classes • Added new
    Module
    storage class, for referencing flows importable from a Python module • New tasks for working with Exasol Big thanks to all our community members who contributed this week 👏👏: • Alex P. • Marwan S. • Peter Roelants • Timo S. The full release notes can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.7
    :marvin: 2
    🚀 22
  • c

    Chris White

    02/11/2021, 10:52 PM
    Hey folks! Data Council is doing a very short survey on what OSS Data tools people are using; the survey is an open response for 3 “open-source data projects” that you’re excited about. They plan to compile and share the results next month — if you’re so inclined, give it a quick look and don’t be shy about including your favorite workflow tool 😉 Direct link here: https://form.jotform.com/210275762605050
    🧠 3
    :marvin: 2
    👍 5
    😄 3
    🚀 5
  • k

    Kingsley Blatter

    02/12/2021, 9:10 PM
    💡Spotlight on a new feature💡 Last night we released a new feature to our UI so that you can override your run config in real time. This is further enhancing the flow run configuration that we added to our core workflow engine last year. For those still looking to upgrade from <0.14.0, see our documentation on the best path forward.
    👍 3
    :party-parrot: 1
    💯 13
    ❤️ 4
    🚀 16
  • j

    Jim Crist-Harif

    02/17/2021, 3:35 PM
    Prefect 0.14.9 is out! 🚀 🚀 Highlights: • Introduced a new
    CompressedSerializer
    for compressing task results • Added configurable timeouts to the Prefect Client • Improved performance for all Prefect AWS integrations And last but not least, Fivetran introduced their first official orchestrator integration with a new Prefect task (the
    FivetranSyncTask
    ); we’re excited to collaborate more with the Fivetran folks so give it a spin and let us know your feedback! Thanks to all our community contributors this week 👏: • Amanda Wee • Andrew Hannigan • Craig Wright • Nick Acosta • Timo S. The full changelog can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.9
    🚀 15
    🎉 10
    🙌 3
  • j

    Jim Crist-Harif

    03/04/2021, 11:22 PM
    Prefect 0.14.11 has been released 🎉 Highlights: • New
    prefect server config
    command that outputs a standalone
    docker-compose.yaml
    file that can be used without the prefect CLI. • New task library tasks for working with files/filesystems in
    prefect.tasks.files
    • Added
    project_name
    /`project_id` to flow run execution context in orchestrated flow runs Thanks to all our community contributors this week 👏 • David Zucker • Jacob Hayes • Joël Luijmes • Timo S. • Yogi Patel The full changelog can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.11.
    🚀 10
    🙌 8
    💯 6
  • j

    Jim Crist-Harif

    03/10/2021, 11:16 PM
    Prefect 0.14.12 has been released 🚀 Highlights: • Complete redesign of the ECS agent to better support per-flow-run configuration and pre-registered ECS task definitions • Two new filesystem tasks Thanks to all our community contributors this week 👏 • Ben Fogelson • Marwan S. • Timo S. The full changelog can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.12
    🚀 15
    🎉 6
    :prefect: 4
  • j

    Jim Crist-Harif

    03/10/2021, 11:55 PM
    Additionally! Thanks to several community members (and our own @Michael Adkins), we have our first official Prefect Server Helm Chart release! 😛arty-parrot: 😛arty-parrot: The chart had been marked as "experimental" for the last few months, but today that flag has been removed. The Helm chart is now versioned, which pins each chart release to compatible releases of Prefect Server, the Prefect UI, and Prefect Core. Versions are published to a helm repo at https://prefecthq.github.io/server/. See the README for installation instructions. On Friday next week (3/19) at 4 EDT @Laura Lorenz will be doing a livestream of how to deploy this helm chart on Azure, see the meetup link for more information
    :marvin: 8
    🙌 3
    🚀 12
    ❤️ 15
  • h

    Hiro Rodriguez

    03/16/2021, 1:25 PM
    Hi everyone! 👋 As some of you know, for the last 2+ months we’ve been piloting a new pricing model for Prefect Cloud and we’ve been pleasantly surprised by the wonderful reception by the community. Many of you have already explored this new way to join Prefect Cloud, through our expanded free tier. With feedback from community members, four ideas became central to what we call “success-based pricing”: 👍 Pay only for what you need  ✅ A generous free tier that includes 10,000 free runs every month (enough to power the majority of small business ETL use cases)  📉 Automatic volume discounts based on your usage  🚀 Committed usage plans for those looking for even deeper discounts or predictable pricing What makes it “success-based”? Pricing is based only on the number of successful task runs processed. We never charge for retries, failures, or even successful tasks that take less than a second to run. And we heard your feedback: all plans come with unlimited concurrency! Whether you’re on a free plan or an enterprise user, we want to make sure that Prefect Cloud is accessible to all in an aligned way. This simple, success-based pricing is available for all new users today, and existing users can choose to upgrade to a free plan from your team settings. Happy Engineering! https://www.prefect.io/pricing
    🚀 17
    🚁 10
    😛refect-cloud: 13
    🙌 4
    :marvin: 18
    👍 2
    💯 23
    🏃 8
    d
    • 2
    • 1
  • j

    Jim Crist-Harif

    03/24/2021, 10:46 PM
    Prefect 0.14.13 has been released 🚀 This release introduces two new CLI commands: •
    prefect register
    for easier registration of flows (docs). •
    prefect build
    for building flows without registration (docs). Combined, these expose a lot of new & interesting deployment options. One is automatic flow reloading. The
    prefect register
    CLI supports a
    --watch
    flag which will watch your source files, and auto-register them on save. This can be useful for development, but also exposes an airflow-like pattern where new flows can be deployed by adding them to a directory. We have several additional CLI improvements in the works, so stay tuned. As always, big thanks to all our community contributors this release 👏 : - Abid Ahmad - Alexandr N. Zamaraev - Jacob Hayes - Timo S. The full changelog can be viewed here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.13 Happy Engineering!
    👍 1
    🎉 6
    🚀 24
    😱 3
    🙌 4
    :prefect: 9
    😍 9
  • k

    Kevin Kho

    04/20/2021, 6:13 PM
    Hello Prefect community! 👋 Join Laura this Friday (4 PM ET) for a livestream about data science on Prefect. This livestream will demo using Prefect in an end-to-end data science workflow, which will illustrate why Prefect is the tool of choice for workflow orchestration. I’ll send a reminder on the day of the event with a direct link as well - hope to see you there!
    🚀 14
    ❤️ 4
    :marvin: 8
    :upvote: 8
  • m

    Michael Adkins

    04/27/2021, 11:31 PM
    Hello everyone! Today we're happy to announce we've released version 0.14.17 🚀 Some highlights from the last few releases: - New
    Git
    storage allows you to use any
    git
    based service for flow storage (this includes deploy token/key authentication!) - Error messages have been improved in more than five places to help guide users to a resolution -
    ShellTask
    now works on Windows and supports more configurable logging -
    Bitbucket
    storage now supports cloud servers - A custom terminal state handler can now be placed on a flow, allowing users to customize the final state of their flow with access to the states of their flow's reference tasks - An architecture diagram has been added to the docs (https://docs.prefect.io/orchestration/#architecture-overview) - The required docker-compose version to run
    prefect server start
    has been lowered and documented - Environment variables can be passed to the
    UniversalRun
    config -
    PostgresFetch
    can return data column names -
    prefect server start
    can now connect to an separately managed postgres instance There's always more, see the full changelog at https://github.com/PrefectHQ/prefect/blob/master/CHANGELOG.md Thanks to our community contributors! - Alessandro Lollo - David Zucker - Greg Lu - James Lamb - Sean Talia - Jonathan Wright - Ben Fogelson - Gabriel Montañola Remember this list isn't exhaustive and for every person listed in it, there are many more who have helped report issues and written PRs that help move us forward even if they aren't merged. Thanks to everyone here in our Slack community!
    🚀 28
    :marvin: 12
    🙌 1
  • z

    Zach Angell

    06/03/2021, 7:36 PM
    Hi everyone! 👋 Today we’re introducing KV Store: managed metadata for Prefect Cloud flows! KV Store allows you to share small pieces of metadata in the form of key-value pairs across your team and flows. You can use it to track useful information and state across your team. Some example use cases we've come across include: 🔢 the last processed value 🤝 easily share state across flows ⚙️ custom configuration Read more about the feature here https://www.prefect.io/blog/introducing-the-prefect-kv-store Please note: you'll need to upgrade to Prefect Core 0.14.21 to get full functionality.
    👍 1
    🚀 17
    ❤️ 11
  • j

    Jeremiah

    06/10/2021, 9:57 PM
    Hi folks -- today we announced Prefect’s Series B! :marvin: :marvin-duck: 🎉 Thank you all for your advice and encouragement - and yes, even your complaints - that helped steer the product and make it as useful as possible. This funding will let us focus on relentlessly delivering value to our users and customers, and we’re excited to do so for years to come. We’re turning everything up to 11, and you can read more about it on our blog! We’ll have many more exciting announcements for this group soon. Until then, happy engineering from our team to yours!
    ❤️ 1
    :prefect: 26
    🤑 5
    🚀 35
    🎉 45
    👏 20
    🤩 14
  • c

    Chris White

    06/17/2021, 7:37 PM
    Hello everyone! The Prefect team will be performing infrastructure upgrades on Prefect Cloud this Saturday, June 19th, between 6:30pm UTC - 7:30pm UTC (2:30pm ET-3:30pm ET). We do expect intermittent periods of disruption during this time window; to stay up-to-date please follow along on our status page: https://prefect.status.io/pages/maintenance/5f33ff702715c204c20d6da1/60cb9ef4ffceea053471149d
    🧰 2
    👍 8
  • j

    justabill

    07/02/2021, 4:29 PM
    Hi everyone! Yesterday, we released Prefect 0.15.0 🚀. This release improves the Prefect Core Python API for running and inspecting flow runs, exposing some powerful new patterns. We've added: • A new CLI for running flows, prefect run, which makes the transition from local execution to running with a Cloud or Server backend seamless • The ability to run flows without an agent, which lets you run your flow locally with the rich set of features enabled by the backend API. • New tasks, create_flow_run and get_task_run_result, to create subflows and retrieve data from them. • New objects, FlowRunView, TaskRunView, FlowView, and TenantView, which enable you to pull the most recent information about flow and task runs from the backend without writing GraphQL queries • A new suite of CLI commands for managing authentication with API keys, simplifying authentication while retaining backwards compatibility with tokens. For more information and examples of how to use these features, check out this blog post. For the full list of enhancements and features, see the release changelog. We're excited to see what you can do with them. If you have a question or feedback about this release, we want to hear it! As always, you'll find us #prefect-community channel. Special thanks to @Stéphan Taljaard and @Tenzin Choedak for contributing to this release!
    ‼️ 4
    💯 14
    🙌 4
    🚀 27
    :marvin: 2
    :cool-llama: 4
    🙌🏻 1
    :upvote: 5
    👍 2
  • j

    Jeremiah

    07/15/2021, 4:13 PM
    This morning, Ben Lorica released a podcast with our very own @Chris White covering a lot of the motivation behind Prefect and starting to touch on the near-term roadmap (more to come on that very soon!). In addition, Ben and his colleagues are conducting a survey on the state of data engineering. Let’s make sure Prefect is well-represented! https://www.surveymonkey.com/r/DEStwitter
    ❤️ 8
    🔥 3
    🚀 16
  • j

    justabill

    07/30/2021, 4:41 PM
    Hey there! We've been happy to see the enthusiasm for the major improvements made in the 0.15.0 release. With the help of some community contributors, we've kept the momentum going with the most recent patch releases, including this week's 0.15.3 release. Cumulatively, these releases include: • Enabled retrieval of secrets from private Vault instances with new
    VaultSecret
    task • Expanded configuration to expose deeper settings for Docker flow runs • Enabled loading of additional repository files with
    Git
    storage • Added documentation for querying account role based access control data • Added
    --no-schedule
    flag to
    prefect register
    to disable flow schedules when registered • Enabled recursive flow registration and builds through globs • Enabled calculation of evaluation parameters on the fly through new
    evaluation_parameters
    parameter for the
    RunGreatExpectationsValidation
    task • Increased robustness of flow runs to connection errors through improved exception handling Special thanks to @Gabriel Montañola, @Nelson Griffiths, Pawel Janowski, @Yueh-Han, and @Chris Ottinger for contributing to these enhancements!
    :marvin: 21
    👏 17
    🚀 21
    👏🏼 1
    👏🏻 1
  • j

    justabill

    09/09/2021, 9:32 PM
    We've been working on some exciting 😛refect: updates that we're looking forward to announcing 🔜. In the meantime, our team and community members have squashed 🐛 and made incremental enhancements in last week's 0.15.5 Release. Here are some highlights: • Python 3.9 support is stabilizing and we are now publishing Docker images e.g.
    0.15.5-python3.9
    . • While registering very large flows, it was possible to see timeouts or incomplete registration. Now, flow edges and tasks are registered in batches allowing for much larger flows. • Agentless flow runs now receive the environment variables of the process that starts them; this makes agentless runs easier to use in production. •
    --idempotency-key
    support has been added to
    prefect run
    •
    prefect run
    will now ignore schedules on local flow runs by default and only a single run will be performed. A
    --schedule
    flag was added if you’d like the command to go forever and run the flow on a schedule. • Several task library improvements have been made, including a new
    SnowflakeQueryFromFile
    task, improved logging in the AWS
    BatchSubmit
    task, improved runtime handling for MySQL Tasks, and updated authentication for the
    DremioFetch
    task. • Log limits have been updated to reduce errors when many large logs are sent. Note that this release includes several breaking changes: • In a secure-by-default effort, services run by the
    prefect server
    cli now listen to localhost by default and are not accessible from other machines; use
    --expose
    to allow connections from a remote location. • Prefect Server will need to be upgraded to version
    2021.09.02
    before flows can be registered from this version of Core. Special thanks to Deepyaman Datta, Henri Hannetel, Johnny Tirado, @Kathryn Klarich, @Tenzin Choedak, and Vincent Xue for contributing to this release :thank-you:!
    :marvin: 7
    🚀 15
    💯 4
    ❤️ 13
  • c

    Chris Reuter

    09/27/2021, 2:56 PM
    Hi everybody 👋! We're going to have a huge week next week. If you want to see the next generation of Prefect tech in action, sign up for our livestream with @Laura Lorenz on October 8th at 4p Eastern/1p Pacific. Sign up on Meetup, or attend the Livestream directly on

    Youtube▾

    . See you there!
    :prefect: 30
    📺 18
    🚀 28
    :marvin: 4
  • c

    Chris White

    10/05/2021, 4:42 PM
    Hello everyone <!here>! I'm extremely excited to finally share what we've been working on the last few months: Prefect Orion, our second generation workflow orchestration tool. If you've ever asked about dynamic tasks, subflows, self-hosting Server without Docker, or almost any other common question over the last few years, the chances are you'll find the solution with Orion.  So much of what we sought to achieve was driven by this community's questions and feedback, and I can't emphasize enough how great it's been to collaborate with you all! Now that this is live, please peruse the docs, test out the initial alpha release and give us feedback!  We want you all to be as excited as we are for the possibilities here and help us shape Orion's near term roadmap. To get started: • read about the vision of Orion directly from @Jeremiah: https://www.prefect.io/blog/announcing-prefect-orion/ • begin building and testing your new workflows with our revamped documentation: https://orion-docs.prefect.io/
    ✨ 6
    :prefect: 23
    💯 44
    🚀 68
    :marvin: 20
    :hero: 6
    🥳 2
    🎉 42
    :party-parrot: 22
    :ahhhhhhhhh: 8
    ✅ 1
    :marvin-duck: 21
    :glitch_crab: 7
    :upvote: 24
    🏗️ 7
    :catjam: 7
    👍 4
  • k

    Kevin Kho

    10/06/2021, 2:35 PM
    Hi everyone, hot on the heels of our Orion announcement, we wanted to get feedback from the community on how you’re using Prefect today, where you’re finding value, and how we can improve. It should take 3-5 minutes. And you’ll be entered in a drawing to win Prefect swag! https://prefect.qualtrics.com/jfe/form/SV_41MC3mkByz18Ok6
    👌 3
    💯 13
    :upvote: 14
  • c

    Chris Reuter

    10/08/2021, 1:05 PM
    Hi everyone 👋! Hot on the heels of the Orion announcement, we have some great livestreams coming up that will feature Orion: -Today at 4p Eastern/1p Pacific @Laura Lorenz will host another spooky October livestream and take Orion for a spin. She'll highlight the new API, subflows and caching, as well as the killer new UI and deployment details. Sign up here to attend! -On Friday, October 22nd at 4p Eastern/1p Pacific Prefect CEO @Jeremiah and CTO @Chris White will host a Fireside Chat focused on Orion. Anyone in the community is invited to join and ask questions about whatever is on your mind. Sign up here to attend!
    :marvin: 9
    🚀 11
    📺 5
  • c

    Chris Reuter

    10/12/2021, 4:14 PM
    Hi all 👋 - more Prefect content coming in 🔥! Scaling Prefect with Dask is always a hot topic in the community. Our friends at Coiled are hosting a livestream with Prefect CTO @Chris White and Coiled CEO @Matthew Rocklin, covering how to scale everything from a single process to an entire auto-scaling cluster. Join them on 10/26 at 6p Eastern/3p Pacific by registering here.
    💥 6
    🐍 11
    :marvin-duck: 12
  • c

    Chris Reuter

    10/20/2021, 2:17 PM
    Hi there Prefect community :marvin-duck:! Coming up in 2 short days on Oct 22 at 4p Eastern we'll be hosting a 🔥 Fireside Chat with @Jeremiah and @Chris White. If you haven't signed up yet you can do so here. The primary focus of this livestream will be Orion🌠, but any and all questions are welcome. If you'd like to submit a question ahead of time you can do so here. If you have any questions live, you can also use the Youtube chat to ask them - we'll be monitoring it throughout the stream. I can't wait to see you all there! Until then, happy Prefecting.
    🌠 2
    :prefect-duck: 3
    🔥 6
    👋 3
    :marvin: 4
    • 1
    • 1
  • c

    Chris Reuter

    10/26/2021, 1:00 PM
    Good morning/afternoon/evening to the best community in the 🌎! Want to scale your Python like a pro? Join Prefect CTO @Chris White and Coiled CEO @Matthew Rocklin at 6p Eastern/3p Pacific to learn how to ⚡supercharge⚡ your workflows. Register here to catch all the hot takes!
    ❤️ 6
    :prefect: 9
    🚀 10
    🙌 1
Powered by Linen
Title
c

Chris Reuter

10/26/2021, 1:00 PM
Good morning/afternoon/evening to the best community in the 🌎! Want to scale your Python like a pro? Join Prefect CTO @Chris White and Coiled CEO @Matthew Rocklin at 6p Eastern/3p Pacific to learn how to ⚡supercharge⚡ your workflows. Register here to catch all the hot takes!
❤️ 6
:prefect: 9
🚀 10
🙌 1
View count: 2