My fear is that this type hinting pushes #python down the same bad road as #cpp:
Making an originally nice programming language more and more complex and ugly by giving in to the pressure of adding more and more features which don't fit into the original idea of the programming language.
In my view, the core philosophy of #python is that it is a high-level dynamically typed language . And the core philosophy of #cpp is that it is a low-level statically typed language.
Don't mix that.
But you don't have to use type hints at all, even in the latest versions. They're completely optional. You could write an entire new application without a single type hint in it, and it would work just fine. Static linters like mypy wouldn't be able to find as many problems without the hints, but that's a tradeoff.
Or does it somehow bother you when some #dependency has type hints in its code?
No, dependencies have their own coding styles (and often different programming languages) anyway and that's perfectly fine.
But "it's only optional" is an excellent excuse for messing with perfect simplicity and elegance.🙃
I get it; I've been a big fan of Python since 1995. When type hints were first proposed it did seem a little at odds with Python philosophy - but I was reassured by the immediate assurances that they were optional, and were going to stay optional. So I mostly ignored them for a few years.
When I did start using them, it felt a little weird; the code didn't "look" right. But that's the same with any change.
Now I'm a big believer. They have prevented so many bugs...
Do you have representative specific samples of bugs you found that way? Trivial bugs or nasty bugs, dangerous skeletons in the closet? Could automated tests have caught them, too?
Did you consider switching from #python to statically typed languages like #java, #rust or #golang ?
There was a time when I couldn't imagine in the world to use a dynamically typed language for serious programming. I changed my view completely. B
May be the same with type hints. But maybe not.😉
@folkerschamel @cazabon @ado IMO, the even if it didn't find bugs (which it does), it helps build confidence in the changes you're making. And any junior dev can throw code against the wall; much of what we do as we climb the ladder is building confidence in the code (or rather helping the junior devs build confidence in the code). (see also: Pirsig's ideas about quality)
To answer the other part of your question: Yes, I'm a big fan of Go. I don't know Rust yet, but I'd probably like it too.
Asking for specific #examples type checking finds is ... unproductive. You write 2 modules of code, run #mypy, and get a list of the 11 places you need to fix - before you've even written unit tests that may or may not have caught the same problem.
If I tried to keep track of them all, I wouldn't have time to write code.
[...]
And yes, many of them are #bugs that would be #agonizing to find by other routes. Your unit test #probably doesn't try passing a #float to that #function you wrote that you only considered using ints with - but with hints, mypy will find that "one weird way" that a float value returned from somewhere else could get passed into the function.
Type hints in code - used well and consistently, no using "#Any" to shut mypy up - make me 50% more productive, I think. </anecdote>
I remember recently learning about one case where some code intentionally mixing integer and float calculations was writing out the wrong type into Elastic Search, causing an inconvenience in Kibana visualizations because integers don't allow the same aggregation functions as floats.
But this is the only case I can remember static type checks probably would have caught, and it was quite harmess.
Maybe I miss something, or maybe our code base is just different, who knows.😉
From the more theoretical perspective - knowing that this is definitely not an exhaustive perspective - right now I cannot imagine situations where a type bug wouldn't be also a functional bug, which would be caught anyway by code review or testing.
Mixing floats and integers comes closest. But also here I think #python does a good job avoiding mistakes. For example, the type of a result of all operations does depend only on the input types, but never on the input values.
@folkerschamel @cazabon @ado I think the main value of static type checking is finding these bugs *faster* and with less effort. I agree that it's not common to have a type mismatch that wouldn't show up as a functional bug under the right circumstances, but code review does not necessarily catch all those bugs, and neither does testing. Or at least, sometimes you have to write a pretty weird test (or expose the program to real live users *gasp*) to find them. I'm envisioning something like a variable which some rarely-executed part of the code sets to None and a different part of the code assumes is always non-None, or passing the wrong number of arguments to a last-resort error-handling function, or stuff like that.
The other main value of type hints IMO is documentation, so that users of a library/function/etc. know what type of thing they're supposed to pass in. I find it a lot easier to understand (and write!) that info in the form of type hints rather than prose.
I see your first point theoretically. But I don't remember situations where I thought "static type checking would have helped us avoiding that trouble". But I keep it in mind.🙂
About documentation, I see the argument in theory. But I think in practice you get that information for free by naming (e.g. "counter" -> int, "name" -> str) or documentation ("number of elements" -> int, "object supporting xyz").
In general, specific real-world examples would be interesting.🙂
Interesting.
Interestingly such cases are not caught by compilers of statically typed languages like #cpp or #java (assuming you are talking about situations where you would use pointers/references in #cpp/#java).
For our software we have the explicit coding style that functions must support None/null/null-pointer parameters for class instances (or more general, never expect conditions met). So maybe this convention helps us creating robust code for such cases.
@DanielaKEngert @tshirtman @cazabon @ado
Interesting.
We have the opposite coding style:😉 a) Parameters and return values must not be references, but pointers, b) function must support null pointer parameters, and c) callers must support null pointer return values.
In our interfaces there are many situations where passing or returning "no object" is quite useful. So to keep it simple and reduce null pointer exception risks, we make it a rule to always support it.
@DanielaKEngert @tshirtman @cazabon @ado
Interestingly it's related to fault-tolerant code.
For example, a function find_some_object (loading a resource) may not find the object, report an error, return null.
The caller should continue, possibly passing that null pointer to other functions without raising an exception or even aborting.
@folkerschamel @DanielaKEngert @cazabon @ado having such an explicit convention is indeed a strategy to avoid this class of problems, but do you have away to check/enforce the convention, or do you have to rely on humans to check it? I assume test coverage requirements might make it more obvious when it's missing, but i have a bias with giving to humans tasks that can be automated, unless the cost is prohibitive.
I also fear this way of handling it might lead to duplicate work in calling/callee
@tshirtman @DanielaKEngert @cazabon @ado
I fullly agree that you should avoid relying on human behavior. Automation is much more powerful. Also, frankly, our automated testing is not as comprehensive as it should be (historical and practical reasons). But in practice at least I don't remember cases where static type checking in #python would have saved us trouble.
Duplication: Yes, our code is full of null pointer/None checks. You see tons of if(obj) / if obj: statements.😉
@tshirtman @DanielaKEngert @cazabon @ado
In #cpp references are the way of disallowing null pointers:
void f(X *x)
versus
void f(X &x)
@folkerschamel @tshirtman @cazabon @ado It goes deeper than that, at least in #cpp
References are (at the language level!) *aliases* of objects. They contribute a *new name* for an object, possibly in a different scope. Therefore it is *impossible* to refer to <not an object>
Pointers are *objects on their own*. They may or may not refer to another object. In Haskell lingo: pointers are *maybe monads*
In some (many) cases, compilers lower them to processors in the same way - if they have to.
Fair point; the code becomes more visually complex. For a beginner, that probably makes a difference.
On the other hand, using type hints saves so much developer time, since bugs are easier to catch, especially with static type checking tools. So adding hints to a project's codebase makes the existing developers more productive.
I suspect that any new dev wanting to contribute to a project will be past the point where hints hurt their participation.
@cazabon @kevin @folkerschamel @ado we're writing an integration for a complex #python behemoth that is spread over many dozens of repos, packages and layers of abstraction. The documentation is insufficient. If they at least had type hints, I would have a much easier time untangling how everything is connected.
Things should be as simple as possible, but not simpler. I hate code where I have to do detective work to see what kind of object I get. I think you're wrong about types.
@zenforyen @cazabon @kevin @ado
#python type hints as "better than no documentation at all!"
Interesting take, didn't think about that.🙂
@folkerschamel @cazabon @kevin @ado that's just to give you a very non-theoretical example.
If you do it right, types are succinct, machine-checkable documentation and safety rail. Because they are optional in python, I can escape them to do dynamic magic that mypy has no chance to follow. It does not have to be perfect to be immensely useful.
And you can avoid creating+maintaining trivial sanity-check tests or isinstance guards in code for things mypy can see and warn about.
@folkerschamel @zenforyen @kevin @ado
Actually, I think that #type #hints in the #function #signature are usually *better* than having them in #docstrings. Not only can they not get out of date / become wrong (without causing checking failures), static type checking is much easier this way.
@cazabon @folkerschamel @kevin @ado even automatically generating nice documentation is better that way. mkdocstrings or pdoc can easily pick the type hints up, make things clickable etc. Without any extra manual effort.
Docstrings should be about the things a tool can't check, not for telling me that it's supposed to be some object following some interface, hidden deeply in a paragraph of text I don't want to read to get that info.
Btw, #Python Protocols are great for lightweight interfaces.
@zenforyen @cazabon @kevin @ado
I see the abstract arguments. But I would love to see a specific example where type hints helped.
Also in our own projects, I don't remember a situation where I thought "yes, we could have avoided trouble here".
In contrary, reading source code like of https://docs.sqlalchemy.org/en/20/core/engines.html#sqlalchemy.create_engine was always more cumbersome than e.g. https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen.
In my experience type hints take away the ease of #python which intentionally makes it different from #cpp, #java, #golang.
@folkerschamel @zenforyen @cazabon @ado This is a small example and a bit of a shameless plug, but check out my jinjanator-plugins project on PyPI and one of the format plugin projects.
Type hints are not mandatory for plugins to that tool, but the API is fully type-hinted and during development this helped me ensure that the plugin code was doing the right thing (returning a mapping of string keys with function values where the functions have the correct shape).
@kevin @zenforyen @cazabon @ado
Do you have a specific sample what kind of problem you caught during development via type hingts in such a way that it saved you trouble or avoided problems down the road later?
Maybe our situation and code base is just different (maybe https://mastodon.social/@folkerschamel/110881640133119780, https://mastodon.social/@folkerschamel/110881535040074008).
Or maybe I miss something and can learn from it.🙂
@folkerschamel @zenforyen @cazabon @ado The problems found by static type checking would have also been found by unit testing, but they were found earlier and with detailed explanations of the type mismatch.
This saved me the time required to troubleshoot failing unit tests.
In this case the API involves a function returning a dict of functions which themselves return dicts, so it's quite easy to accidentally invoke the final function instead of the intermediate one and think you have done it correctly.
@kevin @zenforyen @cazabon @ado
What I definitely see is that static type checking finds bugs at "compile" time before even running your code (whether it's unit testing or just running it yourself). That's for sure an advantage of statically typed languages (and we use for example #cpp a lot). But in practice I don't saw a significant difference here.
Maybe refactoring would be the biggest difference.
Folker
C.
Luke T. Shumaker
David Zaslavsky
Gabriel Pettier
Dani (
Antifascista)
Kevin P. Fleming
zenforyen