I came across dagster (https://github.com/dagster-io/dagster) recently. I would like to now how does it stand in comparison to Prefect open-source offering?

Looks like dagster includes a UI, which is not in prefect core but is in the cloud offering. Prefect Core includes super-simplified scheduling, which I do not see after a quick review of dagster.

I don't have time right now for a full review. @Akshay Verma will you let me know what you find out?

Hi @Akshay Verma, our recommendation for any comparison is to try them both and use the one that you think is best! However, comparison questions are common so I’ll try to give a complete answer here: Prefect and Dagster are both positioned as tools for building data applications. Prefect was designed to solve the negative engineering problem (https://medium.com/the-prefect-blog/positive-and-negative-data-engineering-a02cb497583d), and was inspired by hundreds of real-world lessons learned building Apache Airflow. It was explicitly designed to solve those issues. Dagster has so far failed to articulate a specific problem it solves other than claiming “data is broken.” It provides a DSL for defining DAGs over other tools, like Airflow. Frankly, we are as curious as you to see whom that appeals to. Prefect has always had a philosophy that “Python is the API” and any Python code can be transparently transformed into a Prefect workflow. Dagster started with a very cumbersome, explicit DSL, but has more recently adopted some of Prefect’s ideas, like a functional API, or running on Dask. We’re happy to drive innovation, but bolt-on features are often poor substitutes for well-designed ones. From an integration standpoint, Prefect was designed for every single piece to be completely pluggable via a simple API. This means, for example, that our Dask integration can take advantage of Dask’s data serialization features and resource affinities. In contrast, Dagster appears to have taken a “least common denominator” approach to integrations. This means that while Dagster DAGs can be executed on other engines like Airflow or Dask, they don’t take advantage of any of the unique features of those systems. Lastly, our Prefect Cloud platform forms the backbone of the Prefect ecosystem. It has free and paid tiers to support all use cases. We do not see anything comparable in Dagster; perhaps they will introduce something once we make Cloud available publicly.

hi guys, I am very interested in Prefect to replace Airflow. my company has everything in AWS. there's any recommendation you can make for the Prefect AWS deployment? e.g. AMI ready?

Hey, I’ve noticed that unused allocated memory isn’t freed after a task/flow is run, so if I run a scheduled flow which contains a memory-intensive task, that memory is constantly allocated from the first time the flow runs. I’ve found some workarounds (manually deleting variables at the end of a task’s run function), but this doesn’t always work (e.g. if the output of one task is passed as input to another). Is there a better workaround? I also have a similar issue when running large-scale flows with the Dask executor. It seems like memory is not freed between tasks - I found this relevant issue https://github.com/dask/dask/issues/3247 which suggests that the pool used by a Dask worker to complete a task is not closed after each task. This causes issues with large-scale flows as even if I split the data into small chunks and use these as mapped args to a task, the leaked memory accumulates with each task run and ends up causing workers to die. Has anyone experienced anything similar?

Is it possible to define Dataclass as parameter for the flow?

Hi, At my workplace, we're long-time Airflow users. After skimming through Prefect docs and blogs, I can see that it tackles many of Airflow's pain-points that we've had to hack our way around. But an advantage Airflow provides is the ability to run tasks written in non-Python programming languages, such as R, via operators (Bashoperator, Dockeroperator). Is it possible to do so in Prefect? If yes, how? If not, is it in the roadmap? Congrats and thanks for open-sourcing Prefect core!

Hi, I recently came across Prefect while reading on Dask. Currently I use AWS beanstalk for the deployment of our web application. How can I deploy dask cluster with Prefect in Beanstalk worker environment? Appreciate some help. Thanks

Have you guys used Apache Arrow/Apache Arrow Flight?

IMHO, might be a good option as a return value for tasks...

Dask would probably be the place to implement it, but still...

Just a little comment on the documentation. As an ESL person, when reading https://docs.prefect.io/guide/core_concepts/states.html#state-types section I got confused a bit between state and state types. I'd propose changing it to:

"There are three main state types: Pending, Running, and Finished. Flows and tasks typically progress through these three states types. At each stage of the execution pipeline, the current state type determines what actions are taken.

Sorry if this seems like nitpicking! I don't mean to 🙂

Thank you! This is so valuable to hear — we want to be sure that our docs are easy to follow, and we appreciate the help!

Hi everyone, I came across Kedro (https://github.com/quantumblacklabs/kedro/). Does anyone have insight to how it compares to Prefect Core?

Is it possible to make GUI tool like SAS Enterprice Guide, which can create data workflow interactively and export the final code?

Is there an example of a caching Task Subclass?

I see one with a task decorator.

All of the keyword arguments that you can pass to the task decorator are the same as the initialization keyword arguments for a Task class, so:

@task(cache_for=...)

is the same thing as

my_task = Task(cache_for=...)

(where Task here is whatever actual task class you’re creating)

That’s what I thought, maybe I am calling cache_for in the wrong spot

can you share a snippet of your code?

Unfortunately I can not as the power just went out. But in one of my tests after moving cache_for to the Task construction did not TypeError. I have a couple of tests to run to verify it is caching when power comes back.

Early lunch for me. Power crew down the street didn’t plan appropriately. 🤷‍♂️

This will probably sounds like a weird question, but how is Prefect different from Celery?. I think the differences from Airflow are well described in the docs, yet quite clear, but I'm not sure why I should prefer it over Celery.

Looks like someone’s running long flows watching https://github.com/cicdw/bash-music

Haha nice spot — full story https://twitter.com/jlowin/status/1194088941399478272?s=21

hey, new Perfect user here... is there an HA deployment architecture with perfect and maybe a REST API? or is that something I'll need to craft myself?

Hey prefect people! Just saw that Netflix released Metaflow, their internal workflow engine (ish). From what I can tell, it focuses simply on abstracting away the infrastructure required to run code instead of solving something like the negative engineering problem.I was wondering if you guys were gonna do a blog post or compare/contrast with that product! Just from the 1000 foot POV, it looks a lot like how the documentation for prefect reads for running with prefect’s core. Haven’t dug too much into the details, but would love to hear something because the docs at least read similarly!