RFC for introducing Typing Extensions?

[peterschutt]

Hey @heitorlessa and co.

I noticed this in release notes for 1.25.7:

Later in the year, we'll look into the value-effort ratio of supporting strict mode, and how to gradually bring typing_extensions package across the board.

... and as we spoke about a little while ago I have a some interest in this sort of stuff and think it would be a good vehicle to use for my own learning.

So I have a few questions:

• Has there been a shift in mindset as to bringing Typing Extensions in since we spoke/those release notes were drafted?
• Is an RFC still the way forward?
• Are you looking for help with this?
• If yes, is there a template or an example RFC that I can use as a guide (e.g., this)?
• Are you in a rush? (I don't have a lot of spare time, although maybe this is the kind of thing that shouldn't be rushed anyway 😉)

[heitorlessa]

hey @peterschutt ! Absolutely, answers inline:

Has there been a shift in mindset as to bringing Typing Extensions in since we spoke/those release notes were drafted?

Yes, we should be bringing Typing Extensions.

Is an RFC still the way forward?

Yes. What I'd like to see in the RFC is (1) Typing Extensions as a dev dependency vs runtime (bloat) and (2) What key features we should be using (e.g., TypedDict, ParamSpec, etc.)

Are you looking for help with this?

Yes.

If yes, is there a template or an example RFC that I can use as a guide (e.g., this)?

https://github.com/awslabs/aws-lambda-powertools-python/issues/new?assignees=&labels=RFC%2Ctriage&template=rfc.yml&title=RFC%3A+TITLE

Are you in a rush? (I don't have a lot of spare time, although maybe this is the kind of thing that shouldn't be rushed anyway 😉)

No rush at all :) We're also tied with operational excellence tasks that might take another month or so to complete

Thanks a lot for the help, truly!!

[peterschutt]

Awesome. I'm looking forward to it.

For now, I'll focus on getting a repo setup with an environment for examples, testing, docs etc, and I'll touch base again soon after I've had some time to think about those top-level issues you've mentioned you'd like to see addressed.

[heitorlessa]

Fantastic. Feel free to ping at any time

…

On Tue, 19 Jul 2022 at 11:02, Peter Schutt ***@***.***> wrote: Awesome. I'm looking forward to it. For now, I'll focus on getting a repo setup with an environment for examples, testing, docs etc, and I'll touch base again soon after I've had some time to think about those top-level issues you've mentioned you'd like to see addressed. — Reply to this email directly, view it on GitHub <#1307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBAULCSJLYRCIJPN5QLVUZVLDANCNFSM53ZIQRXQ> . You are receiving this because you were mentioned.Message ID: <awslabs/aws-lambda-powertools-python/repo-discussions/1307/comments/3178806 @github.com>

[peterschutt]

Cheers!

A few early questions:

• is there an easy way to find the current patch versions of the lambda runtimes?
• Have you got a pattern for testing type annotations (otherwise I will prob just try to copy patterns from a project like dropbox/sqlalchemy-stubs)?
• Do you know if there's a way to test against the pycharm type checker? I think it is closed source, right?

[heitorlessa]

Oh I somehow missed this, my apologies:

is there an easy way to find the current patch versions of the lambda runtimes?

Not easily, but we publish the AMI used here along with some info: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Have you got a pattern for testing type annotations (otherwise I will prob just try to copy patterns from a project like dropbox/sqlalchemy-stubs)?

No, I had varying success. I thought of using typeguard but I'd surely want to learn from other projects (sqlalchemy-stubs) as they battle tested this at scale.

Do you know if there's a way to test against the pycharm type checker? I think it is closed source, right?

Closed source yes (last I checked). I'd keep mypy --strict in mind instead. PyRight is improving fast, but I'm not sure I'd use it as a standard.

On that note, there were some great improvements in the latest MyPy change: https://mypy-lang.blogspot.com/2022/09/mypy-0981-released.html

[peterschutt]

On that note, there were some great improvements in the latest MyPy change: https://mypy-lang.blogspot.com/2022/09/mypy-0981-released.html

Thanks for pinging me on that, you are right, there are some really nice features in there.

[peterschutt]

Hey @heitorlessa - just touching base to let you know that I'm still keen to progress this - just happens that my schedule has filled to the brim so I'm battling for time - hopefully somewhat temporarily.

If waiting on me is blocking anything feel free to progress without me.

Otherwise, has anything changed or progressed on the library front relevant to this?

Hope all is well for you and the team!

Cheers.

[heitorlessa]

Hey no problem, we know how it is! The only change since we last discussed is that V2 will drop any dependency (boto3, fastjsonschema, aws-xray-sdk) and make it all optional. This means, if we can't keep typing-extensions as an optional dependency, it won't be a major inconvenience anymore as our final package size will be at ~500K now.

[heitorlessa]

This means, if we can't keep typing-extensions as an optional dependency, it won't be a major inconvenience anymore as our final package size will be at ~500K now

Following up here for the complete picture. We've managed to bring the final package size down to 204K with all dependencies being optional, and we've fully optimized our Lambda Layers reaching 2M final size (even with all dependencies installed!).

We should bring typing-extensions as a runtime dependency and start solving this problem properly.

#1459 (comment)

[peterschutt]

I respect the way you run this project mate, and look forward to working with you. What's the next step then? Time to get the ball rolling?

[heitorlessa]

Just launched V2: https://github.com/awslabs/aws-lambda-powertools-python/releases/tag/v2.0.0

Need a break today :D. I'll create an issue this week to state we'll bring typing-extensions and a list of all utilities so we can divide and conquer. Feel free to pick any you'd like already!

Learned a new trick when trying to do overload for parameters too: https://gist.github.com/heitorlessa/0ff113664e8192fd839dc8c7acfab812

[peterschutt]

Learned a new trick when trying to do overload for parameters too: https://gist.github.com/heitorlessa/0ff113664e8192fd839dc8c7acfab812

Nice! I've used narrowing literals in overloads somewhere along the line before, but the default value stuff is new to me, TIL!

From earlier branch of this discussion:

I'd keep mypy --strict in mind instead. PyRight is improving fast, but I'm not sure I'd use it as a standard.

I've been running both mypy and pyright on my most recent project and we've been doing it in starlite for a little while via pre-commit. I've not noticed a lot of friction (although I think the starlite code base needs an audit of ignores), but I did come across one interesting difference in interpretation of class variable declarations that led me to an interesting read: ClassVar in protocols: mypy and pyright have different opinions

I've had a good opportunity recently to add types to a code-base from scratch: tobymao/saq#43 - I used MonkeyType for the initial types which was quite effective!

[heitorlessa]

That’s super interesting! In your experience, are you finding PyRight a better fit for modern Python?

…

On Tue, 25 Oct 2022 at 01:16, Peter Schutt ***@***.***> wrote: Learned a new trick when trying to do overload for parameters too: https://gist.github.com/heitorlessa/0ff113664e8192fd839dc8c7acfab812 Nice! I've used narrowing literals in overloads somewhere along the line before, but the default value stuff is new to me, TIL! From earlier branch of this discussion: I'd keep mypy --strict in mind instead. PyRight <https://github.com/microsoft/pyright> is improving fast, but I'm not sure I'd use it as a standard. I've been running both mypy and pyright on my most recent project ( https://github.com/topsport-com-au/starlite-saqlalchemy/blob/main/.github/workflows/ci.yml#L30) and we've been doing it in starlite for a little while via pre-commit ( https://github.com/starlite-api/starlite/blob/5656144765ff9fe819b096937186fc2bc13e7c85/.pre-commit-config.yaml#L204). I've not noticed a lot of friction (although I think the starlite code base needs an audit of ignores), but I did come across one interesting difference in interpretation of class variable declarations that led me to an interesting read: ClassVar in protocols: mypy and pyright have different opinions <python/typing#1056> I've had a good opportunity recently to add types to a code-base from scratch: tobymao/saq#43 <tobymao/saq#43> - I used MonkeyType for the initial types which was quite effective! — Reply to this email directly, view it on GitHub <#1307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBGI64CDXBN7OYMCXF3WE4KGFANCNFSM53ZIQRXQ> . You are receiving this because you were mentioned.Message ID: <awslabs/aws-lambda-powertools-python/repo-discussions/1307/comments/3955439 @github.com>

[peterschutt]

are you finding PyRight a better fit for modern Python?

No, I haven't drawn any conclusions such as that. More so, I've just not come across many cases where the tools conflict, and when I have (such as above) its been interesting to dive into it.

[heitorlessa]

Update: Haven't had a chance to write that RFC yet, but, as of the latest release we now have typing-extensions dependency. I've made some good use to launch get_parameters_by_name feature for strict-typing.

[peterschutt]

That one looked like a big PR!

I'm afraid I've not been working on our lambdas for the last little while, so am a little detached from all the changes that you've made recently, but I'll be working on them again soon. As I've not been directly working with the library, I'm not noticing any issues that I can work on, so I'd probably need a bit of guidance as to where to put my attention if I'm going to direct some of my open source effort this way (which I'd like to, nothing changed there!).

One thought I had was that I could look at pyright --verifytypes and use that as a basis for contribution, any thoughts on that?

Also, I'm in the discord server if it is better to discuss stuff like this in there.

Cheers for the update!

[peterschutt]

Had a couple of issues found by pyright and missed by mypy just now.

1. type of __base__ to pydantic.create_model() is type[BaseModel] | tuple[type[BaseModel], ...] - so either a subclass or homogeneous tuple of BaseModel subtypes. I'm passing (BaseModel, MapperBind) where MapperBind is a mixin without any baseclass - mypy lets it slide but pyright picks it up. The pydantic type is really too strict, as it is fine to pass non-basemodel subtypes, so maybe mypy just knows;)
2. I had this del exclude, annotations, namespace where these were functions parameters that I'd transformed and renamed. I removed the annotations parameter to the function, and pyright picked up that I'd left it in the del statement, while mypy let it slide.

Nothing to action, just thought interesting!