@askonomm @zenforyen @cazabon @kevin @ado
Maybe our code base is not large enought😉.
But seriously, I think the reasons are more fundamental.
For example, static type checkers would probably run away in panic from our code base, because we have many situations of indirect function calls manipulating *args.
For example, a cache system based on a #python function based dependency graph, where many function calls are indirect using second order functions and adding parameters dynamically.
@folkerschamel @askonomm @cazabon @kevin @ado I think you're fighting a strawman.
It goes without saying that either you decide to "work with" a type checker, and adopt a #python style to maximize code it can check, or you don't.
Yes, you can't throw pyright or mypy at an arbitrary dynamic wild west code base, and hope it can help you, it can't do magic. So it's a choice you ideally did at the start.
I personally prefer having that additional mechanical pair-programmer in my team.
@zenforyen @askonomm @cazabon @kevin @ado
You can squeeze a round piece into a squared hole. But the solution for a programming problem should reflect the type of problem. Many problems are deeply inherently dynamically typed. I think we have many in our code base.
And when wanting static type checking, then I think it's better to use a static typed language like #java, #cpp or #golang instead of fake static typing aka type hints and misuse #python for something it is not designed for.
@folkerschamel @askonomm @cazabon @kevin @ado If I had the choice, I would not use python for many things.
But there are factors such as library ecosystem and also accessibility. Where I work, still the chance is much higher someone will be able to maintain the type-hinted python code than if I wrote it in a statically typed language.
@zenforyen @askonomm @cazabon @kevin @ado
How would you express something like the following architecture in #mypy friendly #python code or even better in different programming languages like #cpp, #java, #golang etc?
(One part of our software is fundamentally based on countless such calls all over the place.)
def f(smart_evaluator, a, b):
...
def g(...):
x = smart_evaluator.exec(f, someting_extra, a, b)
Of course it is possible, but the question is how natural, easy and elegant it is.
@zenforyen @askonomm @cazabon @kevin @ado
"Where I work, still the chance is much higher someone will be able to maintain the type-hinted python code than if I wrote it in a statically typed language."
For interest: why?
Edit: If I think about it, it sounds like you would prefer a statically typed programming language, but you are forced to use #python against your will, so you are imposing a statically typed programming style onto (poor) #python by using type hints?😉
@folkerschamel
@zenforyen @askonomm @cazabon @kevin @ado
C++20 auto parameters will allow you to elegantly do such things with static type safety. (PS I'm not trying to pit #python or #cpp against each other)
@naresh @zenforyen @askonomm @cazabon @kevin @ado
Over the last 20 years #cpp is getting more and more terrible, unreadable source code. Simply compare #python lambda with #cpp lambda syntax.
I'm open for the challenge for expressing the above sample in #cpp using auto parameters, especially also the implementation of pickling/serializing and hashing the variables a and b.😉
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado
so f is a parameter to the smart evaluator exec call, and smart evaluator is a parameter to the f call? feels like a complicated setup indicating unclear responsibilities.
It's possible to define precisely the signature of f and smart_evaluator, but i'm not sure it's going to make things a lot clearer.
def f(smart_evaluator: SmartEvaluator, a, b) -> SomeType:
…
class SmartEvaluator:
def exec(f: Callable[[Self, …], SomeType], …):
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
What do you mean by responsibilities? smart_evaluator is not owned by f or g.
Yes, obviously you have static types for the smart evualator, but you loose type checking for a and b, which are the main parameters of all the plenty functions, right?
Btw, some other example:
w = create_wrapper_for_handing_over_to_thread_pool(my_object, ...)
x = await w.some_method_of_my_object(a, b)
How can you statically type check a and b?
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
To clarify: There is one SmartEvaluator, but there are hundreds of functions f having very different parameters.
The intention is making adding and calling new functions f via the smart evaluator as easy as possible.
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
By the way, the #chromium #webrtc implementation implements a kind of create_wrapper_for_handing_over_to_thread_pool in #cpp - horrible!!!
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado
I mean that i like my dependencies being a DAG, here you have two things referencing each others, and imho usually that means they need to know too much about each others to cooperate.
Having different parameters to all "delegate" functions makes it really hard to type check statically indeed, if at all possible.
Why not add the functions directly to the smart evaluator class, since you need the instance as parameter anyway?
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado
in the case create_wrapper, it could be a method of te class as well, that returns either a different instance of the class, or setup the instance to be safely called from a thread, and then it's just calling the actual methods of the class, no shenanigans needed to get type checks.
And that's what i mean by unclear responsibilities, if these things are impossible to separate, let them live together in the same class.
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
Wouldn't this mean thatv you have to have to implement every function twice?
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado
Not sure why? Do you want to be able to call f without an evaluator instance? if so your use case is not really clear to me yet (sorry for being a bit slow). And if so i don't see how that's different in your first example, since it was a parameter.
Maybe a real example that makes sense needs a bit more space than 500 characters. 😅
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
With calling twice I was referring to the create_wrapper use case. The idea is that you can easily add or change functions of my_object and get the transition into different threads for free.
Yes, that's not the best place for detailed technical discussions.😉
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
In practice f is always called via an evaluator. But regarding the overall architecture, all the functions f definitely don't belong into the evaluator class. This would basically mean that the whole service would be all member functions of a single class without any modularization, couldn't be extended etc. etc. etc.
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
One case is handing over a async call into a non-async call running in a different thread from a thread pool.
And decorator would break modularization and abstraction between the class functionality itself and the mechanism of handing it over into a different thread (there may be many of them).
In general, there may be many ways. But I think it is not a good sign if you have only a hammer and are forced to treat everything as nail.
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado i don't want to resort to "your system is unnecessarily complex" because it's always too easy to say that without knowing the context and constraints, but i must say it seems you are doing a number of "weird" things :). Surely you have a good reason. So i'll let it go there.
Enjoy your Sunday.
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
Well, I would claim the opposite that it makes our system much simpler than it would be based on a statically typed approach.😉
More in general, these samples are only two of many situations where in my view the problem is naturally dynamically typed. And while it is possible, I would not like to be forced to misuse a statically typed solution.
This is where #python shines.
Many problems can be solved so easily in an elegant compact way.
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
Enjoy your Sunday too, thanks for the interesting discussion!
@tshirtman @zenforyen @askonomm @cazabon @kevin @ado
The smart evaluator class is only a utility class used by (currently) hundres functions f doing the main work and being distributed over many modules, and more modules containing new functions f coming over time.
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado Use typing.ParamSpec and typing.Concatenate.
T = t.TypeVar("T")
P = t.ParamSpec("P")
class SmartEvaluator:
def exec(
self,
func: t.Callable[t.Concatenate[SmartEvaluator, P], T],
something_extra: int,
*args: P.args,
**kwargs: P.kwargs
) -> T:
return func(self, *args, **kwargs)
https://mypy-play.net/?mypy=latest&python=3.11&gist=d3eb03d5b02f194b626aed9021139a55
@bk1e @zenforyen @askonomm @cazabon @kevin @ado
That's cool, didn't know that, thanks!
It seems to me that it works if type variables are used in the same function declaration (your SmartEvaluator.exec) or scope-related function declarations (sample in https://docs.python.org/3/library/typing.html#typing.ParamSpec).
Therefore, am I correct that this solution doesn't work for the some_method_of_my_object situation described in https://mastodon.social/@folkerschamel/110881898235959069?
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado I think your other situation is similar to the `futures` API. The returned object `w` can have a generic type with type parameters that encode additional types, such as the result type. In your case, this would also include the parameter spec.
See https://github.com/python/typeshed/blob/main/stdlib/concurrent/futures/_base.pyi
@bk1e @zenforyen @askonomm @cazabon @kevin @ado
The type of w would have to encode all declarations of all some_method_of_my_object methods of my_object. You could define a interface (base class) for the type of my_object, so that the return value of create_wrapper_for_handing_over_to_thread_pool implements that interface, too.
In general, you don't use duck typing anymore, but are forced to verbosely declare and maintain interfaces and use generics all the time, like in #java or #cpp.
@bk1e @zenforyen @askonomm @cazabon @kevin @ado
All in all, you convinced me that you can probably express all these use cases with type hints in #python.
But it seems to me you are transforming #python into the verbosity and clumsiness (the function declaration syntax + declare and maintain interfaces all the time) of statically typed languages like #cpp or #java (without benefitting from their performance).
Reminds me of the mentioned horror of the #chromium #webrtc #cpp implementation.😉
@folkerschamel @bk1e @zenforyen @askonomm @cazabon @kevin @ado In general, Python with type hints is still *far* less clumsy or verbose than C++ or Java. (I guess I don't really know about modern Java as I haven't used it actively in years, but I can attest to this for C++.) But the type of code I've seen you asking about in this thread is not going to be a good demonstration of that. Perhaps you actually are working with one of those code bases where it really doesn't make sense to use static typing. (which is a valid choice in Python, although personally something I'd take as a sign that there are other things that could be improved about the code)
(in my experience) It's like someone said in an earlier toot: you can reap huge benefits from static type checking, but you have to collaborate with the type system, it's not something where you can tack on type hints to any arbitrary preexisting code base and automatically, necessarily make it better.
@diazona @bk1e @zenforyen @askonomm @cazabon @kevin @ado
If you want/need/benefit from static type checking (which definitely has many benefits), then why not using a language which is designed from that from the ground up?
But in my experience a static type system also has serious limitations. If you have the hammer of a static type system, you can but it may not be the best choice to treat everything as nail.
See https://mastodon.social/@folkerschamel/110898088814600129 where we try to force a cube into a round hole.😉
@folkerschamel @diazona @bk1e @askonomm @cazabon @kevin @ado I think this "vs" discussion is bogus given the fact that python is never planning to force anyone into typing. It's a community decision to either drift towards it, or not, or both will forever coexist.
If it does not look useful to you, just don't use it. Nobody is planning to deprecate hint-less python AFAIK 😉
If both sides could just accept that other opinions exist, we could move on. I think everything relevant was said here.
@zenforyen @diazona @bk1e @askonomm @cazabon @kevin @ado
I think the real underlying discussion is this:
Are there software engineering problems where the natural/easier/shorter/better/cleaner solutions are dynamically typed?
My thesis is: Yes. (Possibly even most of them.) Static typing limits your architectural tool set and can slow you down. So there are costs involved. Sometimes these can be compensated by benefits. Sometimes far from it.
@folkerschamel @zenforyen @diazona @bk1e @askonomm @cazabon @kevin @ado
I'd agree with that statement. So does the python type hinting approach.
We're all consenting adults here. Use hints where you know, you actually are expecting types.
@ketmorco @folkerschamel @zenforyen @diazona @bk1e @askonomm @cazabon @ado Where 'types' is broadly construed, in Python, to mean "something with this shape". It doesn't have to be an actual 'type' (in the OO programming sense), and probably most often shouldn't be a 'type'.
I really wonder how often people use 'list' and 'dict' in their type annotations when they really should have used 'Sequence' (or even 'Iterable') and 'Mapping'. Understanding which of these to use requires fully understanding how you intend to consume the object given to you, and that's software engineering right there!
@kevin @folkerschamel @zenforyen @diazona @bk1e @askonomm @cazabon @ado
Truth. Which is also why I love Python - you can gradually introduce bits of rigidity to your code when you need to. Sometimes it's too soon, and the only ones really qualified to answer that are the ones working on the code base.
I've worked on code where hints are meaningless. And I've worked on codebases where hints would have saved some bacon.
But it's impossible to know ahead of time which is which.
@folkerschamel @zenforyen @bk1e @askonomm @cazabon @kevin @ado ah, gotcha. I was vastly misunderstanding your position; I thought you were making the argument that not only do there _exist_ problems where dynamic typing is the right solution, but that it's the right solution for _all_ problems, at least when working in #Python. I can definitely agree that there are situations where it doesn't make sense to use static typing.
I do still think we disagree on just how often those situations come up, but that's more of a minor difference, and there may not be a productive discussion to be had by hashing it out 🤷
@folkerschamel @bk1e @zenforyen @askonomm @cazabon @kevin @ado Typed Python is still ergonomic, though it isn't the wild west like regular Python can be.
I consider it closer to TypeScrip - it's a lot more flexible than C++ or Java.
@folkerschamel @zenforyen @askonomm @cazabon @kevin @ado I meant something like
class WrapperForHandingOverToThreadPool(t.Generic[P, T]):
async def some_method_of_my_object(self, *P.args, **P.kwargs) -> T:
…
You don't need an abstract base class and multiple implementations. The ParamSpec lets you model the *args, **kwargs forwarding.
@bk1e @zenforyen @askonomm @cazabon @kevin @ado
In this use case are many different some_method_1_of_my_object, some_method_2_of_my_object, all having different statically typed parameters like some_method_1_of_my_object(int, float), some_method_2_of_my_object(string, int)
Folker
zenforyen
Michael Schneider
Naresh
Gabriel Pettier
Brad
David Zaslavsky
Wayne Werner
Kevin P. Fleming
Raniz