I found this talk on Dask interesting and thought the Prefect community might, too. (Even though the title is "Is Spark still relevant?" it's really about Dask.) As Prefect and Dask users, we're very interested in things that drive enterprise Dask adoption: 

Ya’ll have pretty awesome customer service. Just saying.

The new logs UI looks amazing! Nice work!

Typer https://typer.tiangolo.com/ Interesting cli library built on top of click

👀

We use FastAPI / Starlette in Cloud and it’s been great - looks like this is from the same people!

Hello y'all, will the UI be open sourced at somepoint?

unrelated to the product but related to that last comment, thank you guys for drawing my attention to MagicMock() in your incredible tests, saving me hours of frustration :)

will be interesting to see how something like https://www.python.org/dev/peps/pep-0554/ affects Dask if implemented

I did remember hearing on either Python Bytes or Talk Python To Me (can’t remember which or which episode sorry) that it would actually be helpful for their scheduler, but couldn’t replace their cluster computing stuff.

Hey Prefect team, was curious if you guys can share any insights onto how the cloud platform is tested (specifically the GQL interface). I’m building a demo app with GQL for the first time and realized it could be a lot different testing the thing. I’m specifically struggling with getting my head around testing the almost unlimited number of layers of a call. I’ve been thinking about testing the relationships that should be there just to make sure everything works okay. For example against the Prefect API:

query {
    flow {
        name
        task {
            slug
        }
    }
}

vs

query {
    flow {
        name
    }
}

Hi @Alex Cano, great question! 100% agree that there's a lot of surface area to test, here are some tips from what we do: • we use Hasura to manage all of queries. This is helpful because it handles a lot of the grunt work for us, but we still need to test it. To that end, we have tests to ensure each of our entities (for example, flows and flow runs) are properly relating to each other, that they're displaying the proper fields, and that they can only be viewed by the appropriate users. • beyond what's handled by Hasura, we have several additional layers of testing. We test to ensure that we're surfacing the correct fields, that we sanitize and handle erroneous input, that any non-Hasura permissions are correctly applied, our models look like we expect them to, etc. We start small with unit tests for each individual model/piece of functionality, then work our way up to larger integration tests. (And

pytest.mark.parametrize

is definitely our friend) TL;DR Test that fields are being surfaced as expected, test object relationships, and test permissions. Happy to go deeper/broader if you'd like (we have an inordinate number of tests), but these were the items I found most useful when starting out with GraphQL.

Is there any open source alternative to prefect cloud?

So…. anyone read any good blog posts lately 👀 👀

Hi - How does one address the 9.8 User warning directly without suppressing warnings?

UserWarning: DEPRECATED: the result_handler Task option will be deprecated in 0.11.0, and removed in 0.12.0, in favor of the `result` option instead.
  UserWarning,

Hi, I'm having trouble checkpointing a task. Given a task definition:

class SomeTask(Task):
 def __init__(self, **kwargs):
  super().__init__(**kwargs)
  # SomeTask specific init code
 def run(self, input):
  output = process(input)
  return output

And flow:

some_task = SomeTask(checkpoint=True, output_handler=LocalResultHandler(dir=cache_location))

def some_flow():
 with Flow():
  output_handler(input)

if __name__ == '__main__':
 some_flow()

The

cache_location

remains empty and on multiple runs of the flow (with same input to

some_task

) the result is recomputed every time. Could somebody please suggest what am I missing?

Can I get the output of a result_handler on a task from within a state_handler on that same task?

Is there any NO-docker version of prefect to install on-prem (we don't have any docker support)? I did

conda install

and now getting this

No such file or directory: 'docker-compose'

for

prefect server

.

Hello. How can I trigger Prefect Flows programmatically? Is there any Prefect Server API I could call? I'm looking to implement a case similar to one described in https://docs.prefect.io/core/PINs/PIN-14-Listener-Flows-2.html

Truly a random question, but I’d like people who know more about asyncio to chime in. I’ve had experience in path building things using it, which is all fine and dandy. I’m curious if there’s any downside to having everything execute within the asyncio event loop, even if you know things are going to be blocking. Like I know you wouldn’t gain any of the advantages of asyncio, but I was just thinking kind of bigger picture. I’m sure there’s some overhead with running code inside an event loop, but I don’t know enough on the topic to know whether it is significant enough to prevent ^.

I imagine that if you never actually await, there won't be a performance hit outside of getting that first entry point function called by the loop. However you will have much longer stack traces for no good reason and will just generally cause confusion as to what your intentions are?

@Alex Cano your Q reminded me of this slide from a talk I saw last year at djangocon, which plots some empirical data on the case of gunicorn vs twisted requests processed/second on localhost... so a similar situation where requesting localhost/low latency can’t leverage async much at all so the difference is mostly from overhead this chart is from at 31:32. this whole talk goes pretty seriously into async and the whys and whens so the whole thing might be of interest to you if you haven’t seen it already

^^^ yeah we run asyncio http for non-prefect via a multiprocessing setup -- multiple processes, quart instead of flask, etc -- b/c of ^^^. so it matters as soon as you have parallel latency-sensitive tasks. without, something like a pandas call on main thread will kill avg session latency, even if not for an indiv request. but no idea wrt prefect; we were advised to batch tasks to 30s-1min level so don't care as much here.

Yeah definitely. I’ve built out and am maintaining a Fast API app for work, so I’m definitely familiar with trying to keep as much async as possible when you’re already living in async world. It makes you very conscious of your blocking calls and where it’s “correct” to put them. We’ve been using aiohttp, but we’ve heard good things about httpx as well. Mirroring the requests API is really helpful because it can get other devs on board faster

Hi everyone, does anyone know of a way to visualize the relationship between pandas dataframes into a Directed Acyclic Graph? Like

df_B

is obtained from

df_A

by passing it through some function

A_to_B_func()

.

df_D

is made by merging

df_B

with

df_C

etc. If it’s a complex function, yes we can write prefect Tasks to visualize it, but usually it’s an aggregate of small pandas functions and merge, so it would be a pain to wrap each of those into its own prefect task. Maybe I would need to write a complex context manager/decorator or a meta class wrapper around dataframe to record these operations automatically into a prefect task graph? Sorry the idea is not so clear in my head right now…

@An Hoang email leo@graphistry.com -- can get you going w/ https://github.com/graphistry/pygraphistry

can handle pretty fun-sized graphs 🙂

Hi, I see that secrets can be used by anyone within the team. Is there any way of locking down secrets per project or per role?

What's the right way to load auxiliary non-Python files inside of a module? I can use

__file__

and "navigate" from there, but I feel like that's not the right way.