Nash Taylor
05/20/2022, 1:15 AMpip install prefect>=2.0b
) into an environment with mypy, and since there doesn’t appear to be a py.typed
file in prefect
, mypy is skipping it:
Skipping analyzing "prefect": module is installed, but missing library stubs or py.typed marker
I threw a py.typed in there (touch py.typed
), but obviously this is less than ideal. Am I missing something obvious?@task
def call_api(url: str) -> dict[str, Any]:
response = requests.get(url)
<http://logger.info|logger.info>(response.status_code)
return response.json()
@task
def parse_fact(response: dict[str, Any]) -> None:
<http://logger.info|logger.info>(response["fact"])
@flow
def api_flow(url: str) -> None:
fact_json = call_api(url)
parse_fact(fact_json)
This results in mypy complaining about the last line:
No overload variant of "__call__" of "Task" matches argument type "PrefectFuture[None, Literal[False]]"
Now, I have a feeling I know what’s going on here. The @task
decorator causes my function to return, not a dict[str, Any]
, but a PrefectFuture[dict[str, Any]]
, since it’s not actually meant to return data, but state. And in my code, as far as Python is concerned, I’m expecting data and passing in a state.
This kind of hits at a central question I have about Prefect. How are we taking code that operates over data, making minimal changes, and transforming it almost silently into something that operates over states in a way that is still type safe?PrefectFuture
where they are being passed states from other tasks?response["fact"]
gives Value of type "PrefectFuture[Dict[str, Any], Literal[False]]" is not indexable
. HmKevin Kho
05/20/2022, 2:00 AMNash Taylor
05/20/2022, 2:02 AMKevin Kho
05/20/2022, 2:02 AMNash Taylor
05/20/2022, 2:04 AMKevin Kho
05/20/2022, 2:05 AMpyright
is supported but let me see what the status of this is tomorrow and get back to youNash Taylor
05/20/2022, 2:37 AMflow
and task
that define them as pass-throughs whose return type is equivalent to the input type? Given that it seems that the goal of Orion is to stay completely out of the way and allow me to basically just decorate my functions and let me do the rest, I feel like the best way to type that sort of relationship is as though it's an identity functionfrom typing import Any, Callable, TypeVar
FuncT = TypeVar("FuncT", bound=Callable[..., Any])
def flow(func: FuncT) -> FuncT:
return func
@flow
def my_function(a: str, b: int) -> float:
return len(a) * b + 0.5
reveal_type(my_function) # def (a: builtins.str, b: <http://builtins.int|builtins.int>) -> builtins.float
Like... I realize I'm about 3 hours old in the Prefect ecosystem, but if my impression is correct that 2.0 code is basically "pure python plus some task/flow decorators", this feels sort of valid, if hacky?Zanie
05/20/2022, 4:55 PMFlow
class as well.ParamSpec
to capture your types (https://peps.python.org/pep-0612/), but mypy did not have any support for this until recently and I’m not even sure their support is complete now.Nash Taylor
05/20/2022, 4:57 PMUnknown
annotations. But I'm sure I can figure that part outZanie
05/20/2022, 4:58 PMNash Taylor
05/20/2022, 5:00 PMflow
and task
are doing their jobs (via ParamSpec), will the rest even matter much to me as an end user?Zanie
05/20/2022, 5:03 PMNash Taylor
05/20/2022, 7:57 PMfrom prefect import flow, task
because apparently these are "private imports". It asks that I instead
from prefect.tasks import task
from prefect.flows import flow
In a weird way I actually prefer this, but given the documentation always says the other, I thought I'd mention ittasks.py:L231-L255
, and ends up always defaulting to the one that produces PrefectFuture[None, Sync]
. Pyright doesn't have this problem