Nash Taylor
05/22/2022, 10:26 PMreveal_type
of a task run according to mypy: (threaded to avoid an obnoxiously long message)class Task(Generic[P, R]):
def __init__(self, fn: Callable[P, R]) -> None:
self.fn = fn
@overload
def __call__(self: "Task[P, NoReturn]", *args: P.args, **kwargs: P.kwargs) -> None:
...
@overload
def __call__(self: "Task[P, T]", *args: P.args, **kwargs: P.kwargs) -> T:
...
def __call__(self, *args, **kwargs):
return self.fn(*args, **kwargs)
@Task
def my_task(a: str, b: int) -> float:
return len(a) + b + 0.5
reveal_type(my_task(a="a", b=1)) # None
None
it's PrefectFuture[None, Literal[False]]
NoReturn
with None, and swap the order of the overloads...
class Task(Generic[P, R]):
def __init__(self, fn: Callable[P, R]) -> None:
self.fn = fn
@overload
def __call__(self: "Task[P, T]", *args: P.args, **kwargs: P.kwargs) -> T:
...
@overload
def __call__(self: "Task[P, None]", *args: P.args, **kwargs: P.kwargs) -> None:
...
def __call__(self, *args, **kwargs):
return self.fn(*args, **kwargs)
@Task
def my_task(a: str, b: int) -> float:
return len(a) + b + 0.5
reveal_type(my_task(a="a", b=1)) # builtins.float
I get the correct return typeKevin Kho
Nash Taylor
05/22/2022, 10:31 PMNone
, then I get None
. I can't actually tell if that's matching the first or second overload... but in either case, it worksKevin Kho
Nash Taylor
05/22/2022, 10:44 PMNone
is, but I also think that doesn't need to be its own overload, since T
has no bindings and could take on NoneType
. So... yeah, I'm probably just not understanding the purpose of this overload. I'll wait to see what Michael says hahaKevin Kho
Anna Geller
Do you guys ever take time off?we do! 😄 coming week I'll be actually OOO Thu-Sun entirely plus we are experimenting with Slack Office Hours to make it clearer when we are available - see my Slack status
Zanie
NoReturn
was necessary to support functions that return Any
or were unannotated in Pyright. See https://github.com/microsoft/pyright/issues/2142#issuecomment-913249595Nash Taylor
05/23/2022, 4:28 PMState
being a BaseModel
, for example). I'll work up a minimal example to show you the issue I'm havingopenapi.yaml
for a Cloud Endpoints service I'm using.
OpenAPI docs have a lot of keys that use hyphens. The only way I can see to create a pydantic model which can have properties that come out in the .dict()
representation with hyphens is via aliases. For example:
class OneOpenAPIField(BaseModel):
x_google_endpoints: list[str] = Field(..., alias="x-google-endpoints")
test.py
and run it against mypyfrom pydantic import BaseModel, Field
class MyModel(BaseModel):
x_google_endpoints: list[str] = Field(..., alias="x-google-endpoints")
class Config:
allow_population_by_field_name = True
MyModel(x_google_endpoints=["my_endpoint"])
~> mypy test.py
Success: no issues found in 1 source file
~> pyright test.py
...
test.py:11:9 - error: No parameter named "x_google_endpoints" (reportGeneralTypeIssues)
test.py:11:1 - error: Argument missing for parameter "x-google-endpoints" (reportGeneralTypeIssues)
alias
of every field, if it exists. Aliases are almost always used for keys that can't be represented as python argument names, like here with the dashes. So... I basically can't build this model with pyrightfrom pydantic import BaseModel
def alias_generator(name: str) -> str:
if name == "x_google_endpoints":
return "x-google-endpoints"
else:
return ""
class MyModel(BaseModel):
x_google_endpoints: list[str]
class Config:
allow_population_by_field_name = True
alias_generator = alias_generator
model = MyModel(x_google_endpoints=["my_endpoint"])
assert model.dict(by_alias=True)["x-google-endpoints"] == ["my_endpoint"]
assert MyModel.parse_obj({"x-google-endpoints": ["my_endpoint"]}).x_google_endpoints == [
"my_endpoint"
]
.dict()
with an actual TypedDict
Kevin Kho
Nash Taylor
05/23/2022, 5:12 PMpy.typed
to the manifest, so that it shows up in the pip install of prefect>=2.0b
? I'm just adding it myself for now but it would be nice to, y'know, not do thatKevin Kho
Zanie
Nash Taylor
05/23/2022, 5:47 PMNoReturn
overload, leaving me with one overload for async and one for sync. It almost worked. But not really.
• It didn't work at all just by removing the first overload; mypy started telling me that it "needs a type annotation", because @task
was just giving me Any
. I swapped the order of the coroutine overload and the regular overload, and it started giving me the correct type annotation for my synchronous function!
• It gave me Any
for my asynchronous function. Gah.
• 5-or-so hours later, after trying a bunch of things I won't even bother mentioning, I discovered just about the stupidest solution possible: I renamed ___call___
to a regular function name, do
.
from prefect import flow
from prefect.tasks import Task
@Task
def task1(a: str, b: int) -> float:
return len(a) + b + 0.5
@Task
async def task2(f: float) -> str:
return str(f)
@flow
async def my_flow(a: str, b: int) -> str:
task1_out = <http://task1.do|task1.do>(a=a, b=b)
reveal_type(task1_out) # Revealed type is "prefect.futures.PrefectFuture[builtins.float, Literal[False]]"
task2_out = await <http://task2.do|task2.do>(task1_out.result())
reveal_type(task2_out) # Revealed type is "prefect.futures.PrefectFuture[builtins.str, Literal[True]]"
return await task2_out.result()
.do()
instead of ___call___
. Honestly, I would make that sacrifice for the sake of typing, but I'm pretty sure I'm alone in thatdo
(success) to dont
(success) to ___somethingdunder___
(success) to ___call___
(failure)Kevin Kho
Nash Taylor
05/24/2022, 1:35 AMTask
so that calling it was done with .do()
instead of ()
(___call___
), would that change have to be made in a bunch of other places?Kevin Kho
Nash Taylor
05/24/2022, 1:40 AMwait_for
parameter from the interface of Task
, because it's not actually PEP 612 compliant. Check out the rejected case there; wait_for is s
. https://peps.python.org/pep-0612/wait_for
kwarg, I started getting P.args is not defined
and P.kwargs is not defined
.Zanie
submit
function on tasks as an alias of __call__
Nash Taylor
05/24/2022, 2:28 AMZanie
wait_for
and user function types because PEP612 explicit declares appending keyword arguments out of scopeNash Taylor
05/24/2022, 2:29 AMZanie
Nash Taylor
05/24/2022, 2:31 AMTask._call_
(goddammit Slack) takes a P.args
and P.kwargs
, and in pyright it deals with those correctly, but also a wait_for
Zanie
Task
class) would work fine via Prefect Cloud as long as you roll your own Docker imagesNash Taylor
05/24/2022, 2:32 AMwait_for
and P.args
. When I removed wait_for
, it told me I was violating the contract of the supertype.Zanie
do
method and type-ignore your call to __call__
?wait_for
interface that doesn’t require a keyword argument, but there’s nothing quite as simple for users.Nash Taylor
05/24/2022, 2:36 AMwait_for
part of the __call__
signature and not part of with_options
?ParamSpec "P" is unbound
even though the init takes a Callable[P, R]
as a parameter, so I don't know what that's aboutZanie
Nash Taylor
05/24/2022, 3:56 PMZanie
Nash Taylor
05/24/2022, 3:58 PMKevin Kho
Nash Taylor
05/24/2022, 4:00 PMZanie
Nash Taylor
05/24/2022, 4:01 PMZanie