https://prefect.io logo
Title
n

Nash Taylor

05/20/2022, 1:15 AM
Hey guys! Awkward new-guy first question incoming: I just installed Orion (
pip install prefect>=2.0b
) into an environment with mypy, and since there doesn’t appear to be a
py.typed
file in
prefect
, mypy is skipping it:
Skipping analyzing "prefect": module is installed, but missing library stubs or py.typed marker
I threw a py.typed in there (
touch py.typed
), but obviously this is less than ideal. Am I missing something obvious?
More interesting typing adventures: I’m following the First Steps guide with one wrinkle: I basically cannot look at untyped Python code these days, so I’ve typed everything:
@task
def call_api(url: str) -> dict[str, Any]:
    response = requests.get(url)
    <http://logger.info|logger.info>(response.status_code)
    return response.json()


@task
def parse_fact(response: dict[str, Any]) -> None:
    <http://logger.info|logger.info>(response["fact"])


@flow
def api_flow(url: str) -> None:
    fact_json = call_api(url)
    parse_fact(fact_json)
This results in mypy complaining about the last line:
No overload variant of "__call__" of "Task" matches argument type "PrefectFuture[None, Literal[False]]"
Now, I have a feeling I know what’s going on here. The
@task
decorator causes my function to return, not a
dict[str, Any]
, but a
PrefectFuture[dict[str, Any]]
, since it’s not actually meant to return data, but state. And in my code, as far as Python is concerned, I’m expecting data and passing in a state. This kind of hits at a central question I have about Prefect. How are we taking code that operates over data, making minimal changes, and transforming it almost silently into something that operates over states in a way that is still type safe?
Do I need to update all of my type annotations (one so far, in theory plenty in the future) to take in instances of
PrefectFuture
where they are being passed states from other tasks?
Okay well, that didn’t go well either, since
response["fact"]
gives
Value of type "PrefectFuture[Dict[str, Any], Literal[False]]" is not indexable
. Hm
k

Kevin Kho

05/20/2022, 2:00 AM
Not super familiar with mypy myself so I’ll need to ask people tomorrow but the input should be the non-future I think because Prefect will unpack that for you
n

Nash Taylor

05/20/2022, 2:02 AM
https://github.com/PrefectHQ/prefect/issues/5354 Am I interpreting this issue correctly if I say that Prefect basically doesn’t support mypy, but it does support pyright?
It wouldn’t be the worst thing to have to use pyright instead of mypy in this area of the codebase, it’s just a bit annoying since everything else we have relies on mypy checks
k

Kevin Kho

05/20/2022, 2:02 AM
That discussion is Prefect 1 so I don’t know the answer for Prefect 2
And Michael mentioned there the plan is to support it for Orion (Prefect 2.0)
n

Nash Taylor

05/20/2022, 2:04 AM
Bah, I can’t read
I read Github issues / StackOverflow like Billy Beane sometimes. “When you get the answer you’re looking for, hang up”
k

Kevin Kho

05/20/2022, 2:05 AM
Actually no, I can’t read lol. That sounds like you are right only
pyright
is supported but let me see what the status of this is tomorrow and get back to you
n

Nash Taylor

05/20/2022, 2:37 AM
Okay I'm looking into pyright and damn, this might be better than mypy anyway? Honestly if prefect "forces" us to shift to this from mypy, I might not be too mad
Granted, all I've done so far is read the github readme, so I'll circle back when I've actually used it for 5 minutes haha
Sorry, I know it's late, I just got back on this. Would it be sacrilege to write my own stubs for
flow
and
task
that define them as pass-throughs whose return type is equivalent to the input type? Given that it seems that the goal of Orion is to stay completely out of the way and allow me to basically just decorate my functions and let me do the rest, I feel like the best way to type that sort of relationship is as though it's an identity function
from typing import Any, Callable, TypeVar

FuncT = TypeVar("FuncT", bound=Callable[..., Any])


def flow(func: FuncT) -> FuncT:
    return func


@flow
def my_function(a: str, b: int) -> float:
    return len(a) * b + 0.5


reveal_type(my_function)  # def (a: builtins.str, b: <http://builtins.int|builtins.int>) -> builtins.float
Like... I realize I'm about 3 hours old in the Prefect ecosystem, but if my impression is correct that 2.0 code is basically "pure python plus some task/flow decorators", this feels sort of valid, if hacky?
z

Zanie

05/20/2022, 4:55 PM
Hey Nash, we actually are doing a bunch of work to preserve the types of your functions that you annotate.
We can’t just use the identity type variable as you do there because we want you to be able to access all of the attributes on the
Flow
class as well.
Instead we use
ParamSpec
to capture your types (https://peps.python.org/pep-0612/), but mypy did not have any support for this until recently and I’m not even sure their support is complete now.
n

Nash Taylor

05/20/2022, 4:57 PM
I think you're right, I was checking out the issue for it on their repo yesterday and it's still Open
I tried messing around with pyright last night, but I was having trouble getting it to see the installation of Prefect. I was getting a lot of
Unknown
annotations. But I'm sure I can figure that part out
z

Zanie

05/20/2022, 4:58 PM
pyright is the only type-checker with complete support for PEP612 right now so it’s the only one we can use. This sort of typing is quite immature in the Python ecosystem. Our best bet for mypy support is to write a dedicated plugin or wait until they add PEP612 support.
We’re also not running pyright on our source code, so I can’t even guarantee it’s fully working. Pyright is used in VSCode and what I know is working is retention of your types on task/flow calls in the IDE.
n

Nash Taylor

05/20/2022, 5:00 PM
Though how much of the Prefect SDK would I even reasonably expect to see in my code in 2.0 anyway? Like, if
flow
and
task
are doing their jobs (via ParamSpec), will the rest even matter much to me as an end user?
I'm still getting a feel for how much of the 2.0 SDK actually shows up in my code. It seems pretty minimal
z

Zanie

05/20/2022, 5:03 PM
Perhaps not much, but it’s important to have the SDK be discoverable in your IDE.
(Perhaps not yours but our users in general 🙂)
n

Nash Taylor

05/20/2022, 7:57 PM
Okay, it seems like a lot of problems yesterday stemmed from my stupid decision to try to force mypy. Moved to pyright, and things started going much smoother. One minor nuisance, which I'm sure is just a pyright config I have to figure out: it doesn't like when I
from prefect import flow, task
because apparently these are "private imports". It asks that I instead
from prefect.tasks import task
from prefect.flows import flow
In a weird way I actually prefer this, but given the documentation always says the other, I thought I'd mention it
But yes, mypy is unable to choose the correct overload from
tasks.py:L231-L255
, and ends up always defaulting to the one that produces
PrefectFuture[None, Sync]
. Pyright doesn't have this problem