Hacker News new | past | comments | ask | show | jobs | submit login
Python's “disappointing” superpowers (lukeplant.me.uk)
164 points by nalgeon on Feb 1, 2023 | hide | past | favorite | 239 comments



Metaprogramming does not require dynamic types, but this post seems to equate them. As far as I can see all this could be done in e.g. Java (and probably is).

IME this kind of "magic" very quickly loses its appeal when you have to debug it. The author's idea of libraries wrapping this stuff up so users don't have to care about it just doesn't pan out at all in my experience.


Yeah thought so too... in particular the example where you pass a a generator to a db, the generator isn't actually executed, and instead the expression is parsed & transformed to SQL.

This is cool if it works. Good luck when it doesn't due to user error, and more luck if it doesn't due to a bug.

You need to develop a whole strategy to tell the user what went wrong and how, and why, basically from scratch; admitting that you didn't, in fact, execute the generator, and that they have to change it. because of that.

It's pretty awesome that it's possible. And it would be pretty amazing to write something like that. But... idk if it's a good idea from a developer stand point. I like being able to understand what's happening in a library with a single click.


And C# can do that with static typing via LINQ, which is really nice honestly. It’s a bit different but it would be something like:

  var result = await (from c in Customer where c.orders.sum(o => o.price) > 1000 select c).ToListAsync();


> just doesn't pan out at all in my experience.

Fine! These things are based on metaprogramming; chances are you've used some of them.

* In nearly any language, nearly any ORM, for describing tables / models.

* In Rust, serde, and nearly everything you #[derive].

* In Java, Lombok, Hibernate, and most dependency-injection tools.

* In Java, most mocking / stubbing libraries.

* In Python, tons of stuff, from standard library (dataclasses, namedtuples) to various parts of Django, etc.

Judging by the sustained and widespread use of these things, they are not exactly complete failures.


LISPs are the quintessential metaprogramming languages, and they learnt the lesson of "use macros sparingly and only as the last resort" very early on.


I mean, you aren't wrong. But most of what they learned was "step debugging is hard in the presence of macros." And step debugging has largely been tossed out of the window in many modern setups. Just look at Java's "stream" apis. I swear they did what they could to replicate the LOOP macro.


My main problem with metaprogramming in Java is that it breaks automatic IDE refactors and static code analysis. At least the metaprogramming we did with java back in the day (using reflections). If you use common java libraries made with metaprogramming - IDEs usually have plugins to work with that.


I am very rarely concerned about debugging my usage of Java streams, and step debugging works fine in the context surrounding them.


That is also the case for macros in lisp, at large.

Similarly, any heavy use of lambdas in java will make step debugging confusing. As will any annotations you may use. Which is largely why many people grow to hate annotations.

Which is all a fancy way of saying we learn the same lessons again and again. Used smartly, all of these are great tools. Defining what is "smartly" is a place of dragons.


I also strongly dislike the idea that static type systems are only being added to Python because spoilsports from other languages are forced to use Python and don't want to. Not true. That's especially strange considering Python is one of the only traditionally dynamic languages that has mostly led it's own static typing system, joined mainly by just PHP in that regard. Why does it not support features like kwargs? Dunno. TypeScript has no trouble supporting tons of JavaScript patterns you could never do in other languages, even C# from which it is heavily influenced due to its heritage, and TypeScript is a fully separate effort from JS.

On the contrary, static typing in Python is still extremely nice to have. When I was at the peak of my Python career, I had a bug where I changed the return type of a function to be a tuple, and somehow unit tests missed one of the worst possible invocations, leading to an awful failure that occurred after a payment was processed but before actually completing the task, causing it to be retried repeatedly. To be clear, like any failure of this nature, it is one caused by many different problems, and we employed many different solutions; we started paying attention to test coverage, we began using MyPy (it was still quite new; this was also in Python 2 so it needed type erasure compilation among other things) and we made our payment processing logic more robust to prevent processing the same payment twice even in the case of everything else (like retry logic) failing to stop it again. But the thing that sucks is, fairly simple type inference without any additional type annotations could've detected that sort of bug without a potential for false positives.

So I feel like it's silly the way that some dynamic language proponents feel like static typing systems on top of dynamic languages is all from outsiders. On the contrary, the relative weakness of MyPy is actually what ultimately made me quit using Python, and if I had to use it today, then yes, of course I would opt for the highest degree of type safety, as a primary concern above being "pythonic." I like pretty code, but I like correct code more.

Sometimes this does prevent "better" solutions from working, but in my opinion nearly 100% of the time that this is the case, it's because:

- The type system in question is not sufficiently advanced to express the types elegantly.

OR

- The approach is inherently not safe and probably not a good idea. Like patching the request object inside of middleware in Django, for example.

I think TypeScript proves that with a sufficiently advanced type system, even arguably bad ideas can be type safe. For example, the ability to express dotted object paths safely in modern TypeScript is pretty impressive, and lets you map out older JS that does this accurately, but in general that seems like an unnecessary trick that will just make your code slower at runtime for the slightest terseness improvement over alternatives, lacking a language with sufficiently advanced metaprogramming.

The thing is though, eventually this thought process comes true. If a given programming language community winds up bleeding members who are moving on due to the lack of better static type checking systems, then the people left will invariably be much more likely to be against static type systems.


> I had a bug where I changed the return type of a function to be a tuple

> If a given programming language community winds up bleeding members who are moving on due to the lack of better static type checking systems, then the people left will invariably be much more likely to be against static type systems.

This reminds me of one of the things I hate most about the Python library ecosystem: libraries attempt to camouflage type errors resulting from changes like this. Something that should be an immediate runtime error turns into a subtle production bug. I discovered an error of this kind just this week while doing what I often have to do when refactoring someone else's code in Python: reading and distrusting every single damned line.

Which led to reading the source code of a third-party library, where I discovered something that I've seen often enough that I'm starting to think it's some kind of norm in Python libraries:

A function is documented to take a list of Foo. But if the value you pass is not a list, it doesn't throw an error. If the value is None, it uses []. If the value is a set or a generator or a tuple, it uses list(value). If the value is none of those things, it uses [value]. Now that it has a list, it looks at the values in the list, and for any value that isn't a Foo instance, it does its best to handle it. For example, if it's a str, it passes the value to the Foo constructor.

This might feel helpful for beginners and non-professionals just trying to get something to work on a single data set, testing everything by hand, but it amounts to passive-aggressive betrayal when you're trying to maintain a large codebase with multiple contributors. In the case I discovered this week, instead of an immediate runtime error prompting the programmer to make a trivial fix in their code, we had a feature subtly degraded in production for months. Very "helpful."


What's missing here is an Ienumerable from c#. That's basically anything that can be iterated like a list. So a function could take this enumerable and convert it to a list but should probably reject everything else.

And yes generally agree that Paramus should be not be too accepting. It gets confusing fast.


> What's missing here is an Ienumerable from c#. That's basically anything that can be iterated like a list. So a function could take this enumerable and convert it to a list but should probably reject everything else.

Python has an Iterable type class for this.

https://docs.python.org/3/library/typing.html#nominal-vs-str...

https://docs.python.org/3/library/collections.abc.html#colle...


You quit Python because you tried to use static typing in a dynamic language and for obvious reasons that doesn't work.

You could have introduced an UAT environment and done basic QA.


We had dedicated QA testing. QA testing failed to create the requisite conditions.

Type checking is not a replacement for QA testing. However, it is much cheaper and also finds bugs that might be incredibly hard to find through random chance or through structured smoke testing, because it does not depend on how common a given code path is. QA testing is even less likely to find issues than automated test suites, since at least you can qualify branch coverage with test suites (which still will not enumerate all of the possible code paths, and thus... Type checking is incredibly useful.)

So while type checking really is not a replacement for QA, QA is also not a replacement for type checking.

I ultimately quit Python for many reasons, but the desire to write more correct code was the big one. Not just me, but my entire team was burned out on Python. Investing so much into unit testing and increasing coverage, upgrading Python versions, and adding type checking as much as possible was still not yielding the benefits we'd hoped for despite how much effort it took. We eventually found much better success with the then-still-new Go programming language, which we used on most new projects going forward. There was still the occasional head scratcher in production, but deployments were as quiet as a church mouse. YMMV of course; there's plenty of success and failure stories with any programming language. I think we were all ready and willing to put a lot of investment into improving our robustness situation and just felt that we got more out of that effort with Go than Python for our programs. Python still has plenty of advantages too, so we didn't completely quit it; we wound up using Django and SQLalchemy here and there, but definitely most stuff moved to Go over time. (And personally, I stopped using Python for those things too, since moving on.)


I often think that may is just bad. It's type system feels bolted on and limiting. Where typescript can do many things and often teases me into a new pattern, mypy is ugly and stifling.

It's a pity. I really like types but mypy is just not great. It's ok, buts that's not enough for python.


[flagged]


Static typing is not just about code performance, it’s absolutely about correctness as well. I would really love to see that study as it flys in the face of all my experience



> has on average 2.5x the number of bugs

This claim is not supported by the linked article.

In fact, the main claim (only 2% of bugs are type errors) of the linked article also does not make any sense. It is based on the assumption, that typing errors only ever cause TypeError, AttributeError, or NameError in Python, which is, ironically, false, because Python is a dynamic language.

To give a concrete example, I went out to GitHub, got into top Python repository (TensorFlow), and searched for a first closed pull request which explicitly mentioned ValueError and did not mention either of the above. It turned out to be this one: https://github.com/tensorflow/tensorflow/pull/47017 This issue is an obvious type error: in statically-typed languages the value of float can not be nil.

You could probably repeat that experiment with other closed pull requests, but I can bet you you will find that the rate will be closer to 50% if not 90%.


The article says that dynamically typed code and statically typed code has a similar number of bugs per line.

It also says that a software feature only needs 1/2.5 lines if you are using dynamic typing.

That means dynamic typing reduces bugs by 2.5x compared to static typing per software feature.


None of these are anywhere in the linked article: "2.5" "similar"


There is more than one study on the topic. But you can just take a look at the graphs.

Take a look at the "Programming effort" graph and the "Program length" graph.


That not even an error, it's an usability improvement.


Firstly, usability bugs like this are still bugs.

Secondly, think how that issue surfaced in the first place. Somebody used TensorFlow in their Python code, and their Python code produced unexpected result and/or crashed. Judging by the fix, it is highly unlikely to have produced one of the errors from the above list. As a ML practitioner I can also tell you it likely means somebody had to spend a significant amount of time to get to the cause once they saw the output, because numerical miscalculations are very hard to debug in the first place.


That not a bug that static typing would catch as it is not a type bug.


What do you mean it is not a type bug? You've seen the fix with your own eyes. In the static language that fix would never have been needed, because the consumer would never have made the mistake that caused the miscalculation in the first place. Of course it is a type bug!

This is along the lines of

  def is_valid_email(email): return str(email).contains('@')
  
  class Person:
    ...
    def __str__(self): return $"{self.name} <{self.email}>"
  
  print(is_valid_email(Person(name='Peter @ Work', email=None)))


You can't get a NaN from a None value in Python. (Duck typing still has rules)

The error you have posted isn't a Python error.

Looking a bit deeper, the error occurs in C++ code. C++ is a statically typed language, last time I checked.


Not that I am a huge static or dynamic typing fan, but I think language with very strict type system disagree. Like for example Haskell or Rust, where when it compiles, it usually works, unless you introduced a logic mistake. Python with its typing might potentially never get there, but that is not the point.


Do you have a reference to as study that support your point ?


This is nearly impossible to precisely study:

- Because correlation != causation, for one thing: 2.5x the number of bugs in static "style" code bases (not sure what that means) does not mean that static type systems lead to more bugs. It could be that users of dynamically typed languages find themselves needing robust automated tests more often, because they can't rely on static typings. This would suggest that in a similarly productive language, your effort would go much further with types than without, even if this measurement is accurate.

- Because measuring bugs is hard: you don't actually know how many bugs a given piece of code contains, you only know how many you found. Statically typed code contains more information about the intent of the code. Not only does this make it easier for machines to analyze, it makes it easier for humans to analyze, too. Ask anyone who does security audits whether they prefer code that has strict static type information or not.

- Because not all bugs are the same; a reputable study would need to go through great pains to classify the different kinds of bugs and determine their severity as well as determine why they happened. Counting bugs as just a number and then comparing them is like counting SLOC. It doesn't tell you anything on it's own!

> Static typing has nothing to do with code correctness, it's purely about code performance.

Uhhh... Says who? Type erasure systems offer no performance benefits at all, since the types are removed before runtime.

There are contradicting studies, that suggest static typing prevents bugs, but there's little point in trying to fight them. None of the studies I've seen feel robust or complete enough, and on top of that, what would really be helpful is a meta-analysis of multiple more robust studies so we can get an idea of the true takeaways.

Until then, there will be some subjectivity.

There's no nice way to say this: I don't trust people who treat issues like this as black and white. To me, it's a sign of immaturity as an engineer. I do of course believe that static type checking is a net win overall and that the evidence to the contrary likely has other factors (like, for example, relying on relatively primitive type checkers that catch less errors, require more handholding, and generally have trouble handling idiomatic code) but like I said, YMMV: everyone has their failure and success stories. To me, if I can find a bug before I even save the file, versus needing to wait until QA catches it, or worse, it lights prod on fire, it's probably a win. As far as this "static code style" business goes, I don't understand it. My code does not look that different when it's JS vs TS. For Python, MyPy still doesn't seem sufficient, so I would not be surprised if strictly using things MyPy supports would make code worse. For us, switching to Go had more benefits than JUST types; of course we wound up having better performance and memory usage too, and the error handling practices, while verbose, definitely put error handling and edge cases front and center.


[flagged]


> Yeah but typing related bugs which is what static type checkers catch only account for 3% of bugs.

You're stating this as if it's fact and also universal, but I strongly doubt it. Even defining "type-related bug" is hard, because there's a ton of bugs that are not type-related but are much easier to avoid if you're using a good static type checker. For example, stringly-typed values, or switch statement exhaustiveness bugs. Some type checkers, including TypeScript, can eliminate almost all null reference errors when used with strict settings. Then to top it off, types add a bunch of static analysis capabilities you could not otherwise have: for example, you can find accidental dead code because you have an impossible condition that can be proven impossible by types, or more exactly find incorrect usages of library functions, and so forth. There's quite a range of problems and almost none of them are explicitly related to strong typing.

And let's say somehow, this is a universally correct number. That doesn't mean it's the reality faced in front of you. What if you already know the majority of your bugs, as logged today, could be prevented by a type checker?

> You are always going to get more bang for buck by simply writing more unit tests.

Good test coverage is a huge, non-trivial investment for any reasonably large, reasonably complicated codebase. Making all of your code testable in fact impacts how you write code, sometimes in ways that are actually similar to the differences you might make to account for static type systems too.

That said, you should do it! You should do both. Tests catch bugs that type checkers can't, but type checkers offer more than just that, and they catch them faster; typically before you hit save these days.

> Static typing is done primarily for performance and that requires a statically typed language.

I dunno why you're repeating this, people are objectively using static typing for other things. That makes your statement wrong on it's face, because what you're saying is not a matter of opinion, you're suggesting it's "primarily done for performance" and it just simply isn't. Given that TypeScript is one of the most popular programming languages right now, I'd argue this "primarily" categorization is just wrong.

But I think it goes deeper. Like, why does C do static typing? It does not actually need to much. In fact, in an early version of C, the type system was significantly weaker. Struct fields were global: you could access any struct field on any pointer. This is convenient and it doesn't disallow any correct code, but alas nobody is wishing for that to make a return.

I won't even bother getting into functional programming languages, but they exploit types far harder than any of the languages we've been discussing; it's just simply nothing to do with performance.

Maybe C has been trending in the "safer" direction for performance? No. Stakeholders have clearly been improving types specifically for the purpose of preventing classes of bugs observed in the real world. Literally just recently, a great article dropped about array types in Linux, for example:

https://people.kernel.org/kees/bounded-flexible-arrays-in-c

C++ and Rust use types for a lot of things. Some of them are performance, but it's not that simple either. Rust lifetimes are definitely about fast correctness, but it absolutely enforced "correctness" that you would not get with a global interpreter lock, because the principles of never having multiple mutable aliases simply makes sense given the machine model we have today.

So then TypeScript. Let's assume in aggregate it somehow prevents absolutely no bugs despite the obvious fact that it prevents entire classes of bugs that are common in idiomatic JS. Well, guess what? It still offers a ton. For example, it makes refactoring significantly easier. It's not even a discussion, refactoring old code is much easier when it's statically typed, even if it's ossified and weird and full of Chesterton fences. It's night and day. Typescript also offers excellent code intelligence: very accurate autocomplete, inline documentation, automated refactoring options, and yes, more static analysis options than just type checking; being able to statically analyze the code deeper opens up an explosion of linters and checkers for all sorts of things; checking for improper usage of React hooks, or common lodash mistakes.

> Hacking it into a scripting language is plain looking for trouble.

On the contrary, I wouldn't catch myself writing JS without TypeScript anymore. I consider the investment very small compared to the benefit.

In the real world, I often do not get the chance to start new projects from scratch, but instead wind up walking into existing messes. The last JS project I walked into wound up having thousands of errors per day that could've been prevented by static type checking, caused by hundreds of different bugs. And that's just what we could observe by adding metrics: there were still more bugs that were uncommon enough to not be caught initially. When you're dealing with an old ossified codebase like that, you sometimes wind up with more bugs going in than out a lot of the time. The only reliable way I've seen to reverse that trend is by incrementally adding more static analysis. Hope, after all, is not a strategy.

I don't believe there is an objective truth here, but I have strong doubts about the points presented here. The argument for static type checking is strong: it simply prevents classes of bugs, end of story. That's a very simple story and does not require any gymnastics to explain. The argument against static type checking feels very hand-wave-y at best. The fact that we keep landing on this weird point about how it is "primarily" used for performance especially, since I don't think that's true. I think you are mistaking the fact that dynamically typed languages are slow due to the inability of the compiler to optimize for statically typed languages choosing to be static simply for performance. I don't think it's even true for C.


3% is the right order of magnitude and if it is +10% more than that that wouldn't make any difference. Static typing would still be twice as bad as dynamic typing when it comes to code correctness.

"What if you already know the majority of your bugs, as logged today, could be prevented by a type checker?"

You would be a very bad programmer, since the vast majority of bugs are logic bugs.

"why does C do static typing" Code performance. It needs them to effective map types to registers avaliable on the machine it's been compiled for.

Rust lifetimes are about getting C like performance without the bugs introduced by using the techniques required for C like performance. You could remove Rust lifetimes and get all the same safety, it's just the programs would run slower due to garbage collection.

"So then TypeScript. Let's assume " TypeScript is an obvious bad example since JavaScript is a very bad language since it was hacked together by browser makers over several decades. You could make a dynamic typed equivalent of TypeScript and it would have the same advantages.

"The argument for static type checking is strong" It's incredibily weak it prevents a class of bugs that only occur very rarely in dynamically typed languages (3%). By both increasing your development time by 2.5x and increasing the total number of bugs in your program by 2.5x.

You wanting to believe in static typing doesn't stop it sucking in the context of scripting languages.


> You would be a very bad programmer, since the vast majority of bugs are logic bugs.

I don't work alone. Most of the time, I work on projects that predate my involvement, and are old enough to have accumulated cruft.

But I think I found the disconnect. This right here is the key:

> since the vast majority of bugs are logic bugs

You might think that there is no way a type checker could prevent a logic bug. You're wrong. It is clear to me now that when you say 3%, you are talking not about the type of bugs that static type checkers are capable of preventing, but only errors that are explicitly related to types. Because if you model your code with types as you go, types do prevent all kinds of logic errors. When you write code with strong typing, it can prevent impossible state machine transitions, ensure that you handle all cases of a set of possibilities, ensure that you follow API contract obligations, and more.

Exactly how much a given type system is capable of inferring and decoding varies. TypeScript is definitely the state of the art for type erasure systems built on script languages, and what it is capable of encoding is pretty impressive.

Maybe, maybe, the amount of bugs that you can attribute to typing issues is 3% if we're only regarding Python TypeErrors; but, a proper type checker goes far beyond that.


"a proper type checker goes far beyond that"

But if it can't catch 60% of all errors and it can't. Then it's worse than dynamic typing.

This isn't complicated. If you are excluding performance, then static typing is inferior to dynamic typing. It's that simple. No ifs, no buts.

I do write statically typed code but that's for performance critical code. Since I've got good experience of both, I understand the differences very well.

I assure you that using static typing in a scripting language like Python is a mindnumbingly stupid thing to do.


Isn't the C preprocessor a type of metaprogramming?


Yes since metaprogramming is just programming that generates, modifies, or extends other programs.


Strictly speaking, yes. Generally speaking, not really. Metaprogramming implies first class support (i.e., using the language to generate itself)


it's probably discard as being string-based, you could do the same with sed


I kinda hope we can just let Python be Python: it’s in a global-ish maximum for what it’s for, and I suspect any big move away from that will make it worse overall.

Obviously things like UTF-8 support, or maybe getting a clean lexical scope option, or other “fixes” are good. Performance improvements are good.

But there are lots of mainstream languages that have mature static type systems, strong metaprogramming facilities, higher performance, or all of the above.

I’d way rather see effort go into getting the package management thing figured out, or the static analysis stuff improved further than any big movement in the language proper.

Just my 2c.


Nit: Python has strong metaprogramming facilities since version 2.2, when "types" and "classes" were made equal, and it became trivial to create a class using the `type` function. It's so easy exactly because Python is so dynamic.


I still think python has a ton of headroom to grow. It's maxing out the data science space, and it does reasonably well in the web service space, but they really have not pushed very far into the distributable GUI and CLI space. Dependency, packaging, and distribution have been improving in leaps and bounds still for python, and type-hinting makes large projects manageable.


I'm a so called "scientific" programmer, not a software developer. My work is upstream from turning anything I make into a product by the real devs, who have their own languages.

Maybe I'm in a minority, but I use dynamic typing. I can haul data around in flexible formats such as JSON, asdf, pickle, and so forth, and manipulate them as dict structures. Python builds the structure automatically.

One of my first languages was Pascal, and so I understand the principle of letting your data structures and type checking do your work, and I still live that way when programming in C for embedded stuff. There may be better ways of doing what I'm doing, but there are also worse ways. From my own experience with getting software changed to suit my own development work, data structures and file formats are right up there in terms of why software is so inflexible.


> I’m worried that a de-facto move away from dynamic stuff in the Python ecosystem, possibly motivated by those who use Python only because they have to, and just want to make it more like the C# or Java they are comfortable with, could leave us with the very worst of all worlds.

It is certainly happening, and I'm not sure Python the language is all the better for it. I say that as a guy who explicitly mentioned this "Python Is Not Java" blog-post during my first job interview as a professional programmer, more than 15 years ago.

[1] https://dirtsimple.org/2004/12/python-is-not-java.html


Over the years C# (and probably Java, but I'm not so well versed in that) gained quite a few features to improve "Meta-Programming", one could argue in an attempt to compete with dynamic languages. In C# that is mainly possible through introspection, generics and maybe dependency injection. Entity Framework Core can mostly figure out an SQL Table and its queries by looking at a simple class. It's not as good as the Django ORM or SQLAlchemy, but it tries to achieve similar things...

So maybe some stuff in the Dotnet ecosystem got started because Python programmers were forced to use C# and tried to make that more dynamic.


How is SQLAlchemy or django better at any of that? in my experience EF is far far superior


Migrations have been painful in EF. No easy way to separate models/migrations for the same db context like with Django apps. Metadata for the models is hard to come by (probably there is a way?). Support for advanced Postgres datatypes slowly becomes ready for primetime, but currently isn't, and it's certainly about ten years behind Django in that regard.

In Django, a model alreay contains all the meta data and you need almost no code to generate CRUD views and APIs from it. Can't do that in Dotnet. I think I could make something like that but why bother and take the pain?

When dotnet people say "my experience" then they mostly mean no experience beyond dotnet (in my experience). I hope you are basing your comparison on more than a glance or a weekend project trying out Django...


> Support for advanced Postgres datatypes slowly becomes ready for primetime, but currently isn't, and it's certainly about ten years behind Django in that regard.

What type support is lacking in the ef pg provider?

> Metadata for the models is hard to come by (probably there is a way?).

I'm not sure what metadata means here exactly, you can configure models with either the fluent api or the attribute api


Json(b) columns, (multidimensional) Arrays.

By metadata I indeed mean info on what a field means: Human readable names, descriptions, hints on validation or user input fields. You can argue that's not the ORM's job, but having some of that already in the ORM allows Django to have CRUD apis and views with very little code beyond the models.

I think you could make something like DRF or Django admin on top of EF by abusing assembly introspection. But so far nobody dared to do that.


Check the history of IronPython and IronRuby and how that spun the Dynamic Language Runtime, likewise do the same for jython, jTcl, jRuby and invokedynamic bytecode.


My theory is that Python and C++ are slowly evolving towards eachother, and in 20 years will merge into the same (very confusing) language.


C++ consuming whatever popular paradigm is around and growing into a jumbled mess? Surely you're joking...


C# and typescript towards each other for sure,

And everything taking from lisp/other functional languages: lambda, map, filter etc. Which I really like


eval() makes it harder to reason able your code and opens you up to injection attacks. Steering the boat toward C# or Java is better than crashing into TCL.


I think type hints have mostly changed Python for the better but I still get frustrated by the number of half baked features and inconsistencies in the language. You end up fighting quirks ( like isinstance not working properly with generics ) all the time and it can get pretty tedious.


... what? You're not supposed to use isinstance with generics??? It's a type-hinting only feature afaik


> But typing.Callable has zero support for them, meaning they can’t be typed in a higher-order context.

https://peps.python.org/pep-0612/

    from typing import Awaitable, Callable, ParamSpec, TypeVar

    P = ParamSpec("P")
    R = TypeVar("R")

    def add_logging(f: Callable[P, R]) -> Callable[P, Awaitable[R]]:
        async def inner(*args: P.args, **kwargs: P.kwargs) -> R:
            await log_to_database()
            return f(*args, **kwargs)
         return inner

    @add_logging
    def takes_int_str(x: int, y: str) -> int:
        return x + 7

    await takes_int_str(1, "A") # Accepted
    await takes_int_str("B", 2) # Correctly rejected by the type checker
I think the author and the people working on Python's type annotations actually agree here. The goal is to have type annotations powerful enough to statically describe all behavior generically without library specific extensions. There hasn't been much push for it but I expect a lot of the author's concerns would be alleviated by a type annotation for types themselves. Being able to say, "alright I'm gonna so some dynamic metaprogramming magic bullshit, but at the end will pop out a class of this shape" probably gets you 80% of the way where.


When I did most of my work in Python, typehints weren't a thing. I still feel like static typechecking is less useful in Python since its datamodel is overally less of a hot mess compared to Javascript for example.

And Python modules/frameworks can be a lot more "typed" than you'd expect. Django models, forms etc offer more runtime validation than typesystems can easily achieve. Yes, at runtime. But between not having a build process and in general being more readable and concise, I feel like Python lets you run into the runtime error long before you get the same thing done "error free" in C#. Not that C# doesn't have runtime errors...


> its datamodel is overally less of a hot mess compared to Javascript for example.

I don't understand this, what do you mean? TypeScript is such a success partly because everything is an object in Javascript. There's no need to distinguish between classes and "dicts", as you have to with Python. Another example: Python functions have args and kwargs, which make typing significantly more complex (as the article points out, Callable doesn't even support kwargs). Javascript doesn't, so typing functions is trivial.

Part of the point of static typechecking is documentation that can be verified by tooling. If you have no types in your codebase, and you decide to change something from a dict to a class, good luck! You're going to have to manually trace all the data flows through your entire application to figure out what's affected.


> everything is an object in Javascript

Everything is an object in Python too.

> There's no need to distinguish between classes and "dicts", as you have to with Python.

"dict" in Python (or at least Python 3, but Python 2 is EOL now so Python 3 is the only active Python there is) is a class. You can even subclass it, which if you want a customized dict for some reason in your particular application is often the best way to do it. Or you can subclass MutableMapping, which dict itself is a subclass of, and which is what is supposed to be used for isinstance checks now that Python has established abstract base classes for all of its standard "built-in type" interfaces.


> Everything is an object in Python too.

Ah yes, true. JavaScript goes one step further though, and has no distinction between get-item accesses and attribute accesses. This makes TypeScript's data model much simpler to manage. Is it even possible to implement your own `MutableMapping` or `dataclasses` equivalent type in Python? I know the latter requires custom plugins.


> JavaScript goes one step further though, and has no distinction between get-item accesses and attribute accesses.

Can you explain a bit more about what you mean here?

> Is it even possible to implement your own `MutableMapping` or `dataclasses` equivalent type in Python?

Of course. With dataclasses that's what Python itself does; that's a pure Python module:

https://github.com/python/cpython/blob/3.11/Lib/dataclasses....

For MutableMapping, Python currently implements it as a C class for speed, but you could implement the same functionality in pure Python. That's what earlier versions of the collections.abc module did.

> I know the latter requires custom plugins.

I don't know what you mean here. See the pure Python module that Python itself provides above.


There is no difference between `object["hello"]` and `object.hello`. So in Typescript, all you need is `interface`, that's it. Everything (except primitives) is just an `interface` with keys or index signatures.

> Of course. With dataclasses that's what Python itself does; that's a pure Python module:

Sorry, I meant that you can't create an equivalent class that typechecks the same way. For example, when you define a dataclass with fields, mypy and pyright are capable of inferring what args and kwargs the constructor supports. But that's because it was implemented as a special case. For example, see the mypy docs [1]. If you just alias `dataclasses`, you lose type support!

Pydantic is a library with similar semantics, and the only way to get typechecking support is to install the pydantic plugin for the typechecker you use. The language simply can't express it.

[1] https://mypy.readthedocs.io/en/stable/additional_features.ht...


> There is no difference between `object["hello"]` and `object.hello`.

Ah, ok. Technically I suppose one could unify these in Python; attribute accesses on objects are really dict lookups under the hood, and one could implement a list as a dict whose keys were restricted to be integers. But yes, Python chose not to go that route.

> I meant that you can't create an equivalent class that typechecks the same way.

Well, of course not. If you derive your own class direct from object, even if it's duck type compatible with some other class, it is not a subclass of that other class. That's just how Python's type system works.

> If you just alias `dataclasses`, you lose type support!

You mean you lose some functionality with third party type checking tools. Yes, that's true. (Although that particular limitation, at least, seems odd to me--see below.)

> The language simply can't express it.

No, the language does not have built-in tools that do the things you would like them to do with type checks and type annotations. And the third-party tools apparently don't cover these cases.

The "alias dataclass" case seems odd to me because a simple "is" check ("dataclass_alias is dataclass") should cover it--aliasing the object just means putting a reference to the same object in another namespace, and the "is" operator checks for object identity, not the name it has in any particular namespace.

The "dataclass_wrapper" case would be a good bit more complicated to check for, yes. Although type annotations ought to be able to help, since you can express "this function takes a class as an argument and returns a dataclass derived from that class" in Python type annotations.


> You mean you lose some functionality with third party type checking tools

No, I mean that Python cannot express this relationship with type annotations. That's why plugins are necessary, because the language doesn't support it. I think the same is true for `TypedDict`, which is why the community had to wait for official support before it could be used.

These fundamental limitations of the language prevent the community from effectively building new tools with similar behaviors. You have to either write your own plugin for the n different 3rd party type checking tools, or hope it gets pulled into an official Python version (at which point the n different type checking tools will implement support for you).

Anyways. This started off because of the claim that Python doesn't need typechecking as much because it has a simpler data model than JavaScript (at most, you can claim they are equivalent), but it's turned into gripes about how half-baked the type annotation syntax is. All of this to say: it could have been a contender! It could have been like TypeScript!!


> Python cannot express this relationship with type annotations.

What relationship? That two classes which have no ancestor classes in common happen to be duck type compatible? Yes, Python type annotations can't express that, because there is no type relationship to express.

The way to avoid this problem in Python is to make sure duck type compatible classes have an ancestor class in common. That was a main point of the collections.abc module when it was introduced, to provide actual classes that expressed the duck types that were built into Python (being a subclass of MutableMapping, for example, expresses the fact that a class is duck type compatible with the built-in dict class).

As for "dataclass", it's not a class, it's a function. But the dataclass module provides the "is_dataclass" function that you can call on any class to check whether it's a dataclass. As far as I know, this will detect both your "dataclass_alias" and "dataclass_wrapper" cases. So this function could be used by any tool that wants to check for dataclasses.


> All of this to say: it could have been a contender! It could have been like TypeScript!!

No, it's to say: "I think Python should have done things the way TypeScript does!" But Python is not TypeScript. It's a different language with a different design philosophy. Expecting to use it exactly the way you would use TypeScript is of course not going to work out well, just as expecting to use TypeScript exactly the way you would use Python is not going to work out well.


> I think the same is true for `TypedDict`

The typing module is pure Python, so the code for TypedDict in that module could have been implemented by any Python user. The official support is of course nicer, but is not required to extend the functionality of the typing module.


> Python functions have args and kwargs, which make typing significantly more complex

Not really. It's only complex when they're used with poor design decisions. There are plenty of languages that allow you to do things that aren't a good idea. Python is no exception. Type hinting just makes those shortcomings more readily apparent.

> Callable doesn't even support kwargs

PEP 544 and Protocols have existed for 5 years, and they support keyword only type hinting for callables.


I'm talking more about really simple patterns that Python's type system just can't support. Take kwargs. They've existed forever. A really common and not-stupid pattern is to write a subclass such that you only specify some arguments you care about, and then take *kwargs to pass on to the super constructor. Can't type this. You have to exhaustively enumerate those kwargs and pass them in manually, or else you lose type hinting for your subclass.


What you call a problem is simply how constructors work: users pass in the arguments that are expected for a certain way to initialize the object, and you either document these expected arguments with type hints or "lose type hinting".

Expecting to inherit type hints from an unrelated place, like the constructor of a superclass, is quite unjustified. The superclass is an implementation detail.


Yes, that exact scenario is supported with TypedDict.

And while that is a simple pattern, it's not a good pattern, and quickly becomes unmaintainable in a program of any size. Using *kwargs because of Too-many-arguments and so you can violate the Liskov substitution principle isn't good programming. Just because it can be done doesn't mean we should start bastardizing typing to make it easier.


> Yes, that exact scenario is supported with TypedDict.

I don't think so. I just threw together a simple example that doesn't work:

    from typing import TypedDict

    class A(TypedDict):
        name: str
        age: int

    def myfunc(**kwargs: A):
        print(kwargs)

    myfunc()
No errors with mypy. In fact, kwargs is inferred as `dict[str, A]`, which is unbelievable given that Python explicitly supports `**kwargs` as a way to represent a grab-bag of keyword arguments.

> And while that is a simple pattern, it's not a good pattern, and quickly becomes unmaintainable in a program of any size

It's not though, if you had a proper type system. For example, you could do this trivially in Typescript and it would automatically infer the types of the rest of the kwargs. That means that your type information is preserved, making it possible to keep using your subclass without losing information about the total set of arguments it supports. The advantage is that you can add/remove arguments to your base class without having to hunt down every subclass, and you're guaranteed to be correct (and it would error otherwise!).


With hot mess I mean things like undefined vs null, various surprising comparisons, truthiness, lack of a primitive integer type...


OP is not saying that type checking is useful. They’re saying that it’s not as useful.


What about Python's data model makes typechecking less useful? You still need to know how data flows through your application. You can either document it in your types, or require your engineers to hold all that information in their heads/recapitulate it every time they want to refactor code. That has nothing to do with the language's data model.


[flagged]


Citation heavily, heavily needed. Dynamic typing means that none of your code can be statically checked, which means that correctness relies on the exhaustiveness on your tests. It should be obvious that this would allow you to introduce more bugs, not fewer.

Every program involves passing around data. When you need to change that data in a dynamic untyped language, you have no option but to manually test your whole system to make sure that you nailed all the spots.

Compare that to a language like TypeScript. If I need to refactor my code, I update the type and the static typechecker immediately tells me all the spots I need to fix. I don't have to hunt for them, I don't have to run it 100x trying to hit every edge case. And JavaScript is still dynamic, it doesn't stop me from writing concise code.

This isn't theoretical. I literally just worked on changing something from a dict to an object in Python, and it was a pain. Thankfully most spots were documented with type hints, but even then I ran into many runtime errors that could have been caught statically.


There is almost no evidence type safety helps at all. Anecdotally, I don't find I have bugs in Python that could be caught by typesystems easily.


There are numerous static analysis tools for Python that predate type annotations, the biggest of which are pylint and pyflakes.


[flagged]


> No citation is needed, it's fairly obvious to any experienced Python programmer.

So you can't even explain it, despite claiming to be experienced. Why should I listen to you? Your whole response is just you saying "First off, you're wrong. Second off, I'm right."

Here's a very simple example: I want to change a dict to a class. In untyped Python, how do you make sure that you make this change safely without introducing new bugs into your application?


Here's a citation if it makes you happy: https://games.greggman.com/game/dynamic-typing-static-typing...

2% of bugs can be caught via static typing.

For your very simple example, your IDE can do that!


Here is a study that show Python is more bug-prone and that TypeScript is far more reliable: https://danluu.com/empirical-pl/

That source shows why language studies are so difficult, and why it is hard to draw conclusions. For example, your study doesn't count KeyErrors or NoneType exceptions as type errors, but those count as well. There are other considerations: you're only looking at released projects which might have a ton of tests to balance the fact that they're dynamic. But there is absolutely no way you can say "dynamic languages are less buggy". There's no data for that.

> For your very simple example, your IDE can do that!

No it can't. An IDE doesn't help at all here, because it doesn't know where you passed around that dict, and where it was created. There's no type hints. That dict is now a class, and you have to go find where else you used it. If you don't use any type hints in Python, how would you find all those cases? You have to hunt for it yourself, grok the entire codebase (which you may not have worked in for a while), and then run it a ton of times to make sure you caught every edge case. Refactoring in Python is very time consuming and dangerous compared to typed languages.


[flagged]


That is the most brazenly false statement I've ever heard. "Python is superior due to dynamic typing, and also all types in Python can be statically inferred!" Wow, you should let them know that type hints was a total waste of time because you could already infer them!

I think it's clear you have a very weak understanding of this topic, and you have no interest in having a civil conversation. Good night!


You have clearly made up your mind and no matter what someone says you won't change it. Good luck learning new skills.


Well I'm showing evidence for my position and no one else is. My mind could be changed with evidence. But there isn't any as I'm correct.


Either embrace dynamic typing and provide good error guards...or try to use type hints and still make good error guards.

We had an entire history of Python 2 without type hints. Why use them now?


How come everyone realised the benefits of typing by now, except the Python community?


Because python got a half baked, overall poor quality type hinting system so devs that haven’t been exposed to the benefits of eg. Typescript don’t understand why this is better.

Many of us in the python community know the benefits, and we’re simply changing our stack so python is as contained as possible, and dropping lower quality libraries for higher quality typed ones.


The benefits of typing for certain kinds of applications.

Many people in the Python community simply aren't writing those kinds of applications. They're writing applications where typing is not a benefit, it's a hindrance, so they don't use it, and Python makes that easy.


Could you give an example of a type of application where typing is a hindrance?


I did a fairly detailed breakdown regarding the Python library Parsy: https://lukeplant.me.uk/blog/posts/python-type-hints-parsy-c...

This is not to make the general claim "Typing is a hindrance in parsing applications", or anything close. It's saying "the current static type system(s) available in Python would have made this Python library much worse".


Not a hindrance but I work with image data a lot. Usually what comes in is any kind of Numpy array and what comes out at the end is either statistics or a Numpy array. I try to type hint everywhere I can but I feel stupid doing so because all I do is work with arrays. Maybe I’m not proficient/knowledgeable enough though so take it with a grain of salt.


If it's truly "any" kind of Numpy array typing won't help much there I agree.

But often it can be helpful to know the shape of the input and the shape of the output. Eg are we working with a 2D,3D or ND array.

Before 3.11 it was not simple (possible?) To type this. But https://peps.python.org/pep-0646/ makes it possible. If somewhat clunky on the definition side.


I’m also part of the Python community and I use typing extensively. I find them useful for app development but also in scripting. There must be at least be a good dozen of us seeing the popularity of mypy, pyright and the fact that the language designers saw fit to add them, too.


Not to be rude, but scientific/ai/data scientists are typically not the best "programmers" (best practices and code hygiene) , and their stuff is the main reason to be on Python these days (other than for small CLI and scripts), which pulls everything around into it like a black hole ("oh let's make the api in Python too, to keep it one language").

And Python has a ton of newbies, college kids etc. Though this is also true of JavaScript.

As an industrial language/environment with best practices for very large codebases, it's really weak and I much prefer TypeScript despite the occasional tooling fatigue (has slowed down).

But a ton of money has been poured into JavaScript (v8) which really set that into motion.


[flagged]


> Developing with static typing reduces development speed by 2.5x and increases bugs by 2.5x per the emperical studies on the topic.

I hate "Citation needed" comments, but this is a very strong claim and so deserves it:

Citation needed


[flagged]


> It's not a very strong claim, it's due to dynamic typing enabling the writing of more concise programs. The number #1 bug elimination tool is to simply write less code.

It is a strong claim, it's also a very precise claim. Where do your numbers come from?

Haskell and the ML family of languages also permit you to write much more concise programs than many other languages and they're definitely statically typed. How do they fare in your imaginary analysis?

> As a matter of principle, I won't provide citations unless you first provide some citation that says I'm wrong first. (You won't find one)

Ok, so you're an idiot. Got it. You can't make such a strong and precise claim and not back it up, and try to turn the tables by saying "No your citation!". You just come off like a petulant child (maybe you are?). Your claim makes no sense, and anyone who has at least 5 years of experience (probably less, actually) would know that.

You've asserted there are empirical studies on the topic to back your numbers. Just provide them.


[flagged]


This argument is in such bad faith it's hilarious.

Here's my much better argument: Instead of using dynamic typing, eating babies reduces the incidence of bugs in code by 1000x. If you don't believe me and would like me to cite a source, you first provide a citation saying something different to what I'm saying and I'll provide my citations. Come on, you can do it! (Hint: You can't)

If it really was a strong claim, you could disprove it easily.


And yet I eventually provided evidence for what I'm saying and the other party provided nothing at all.

It's not bad faith because you don't like the arguments conclusions.


I support you ReflectedImage - seems like these people just don't know how great dynamic languages can be! My team has actually seen a 30x decrease in bugs, and a 50x increase in productivity ever since switching to Python.


Well a lot of people are claiming that static typing reduces bugs compared to dynamic typing, but I haven't seen from them a shred of evidence of this.

When I go looking for myself what I found was studies saying that dynamic typing reduces development time by 2.5x and bugs per software feature by 2.5x.

So perhaps one of these people claiming that static typing reduces bugs over dynamic typing, could provide the slightest bit of evidence to backup their claims?

Is that too much to ask @spoils19?

But I'm pretty sure they will just find the exact same figures as me.


In the amount of time you’ve repeatedly said this all over this thread you could have provided a citation


Here: https://games.greggman.com/game/dynamic-typing-static-typing...

I think it's more effective to show that there is no evidence to contradict me first.


lol wow you're really something, this is literally one guy, there is TONS of research on this which contradicts his findings. This guy does a pretty good summary https://danluu.com/empirical-pl/ definitely heavier on the research wouldn't you say?

He also misses the biggest add of typing your code which is that other people can understand it better. This is often lost on junior programmers...


Well according to most of that research I'm correct.

See: "Work In Progress: an Empirical Study of Static Typing in Ruby; Daly, M; Sazawal, V; Foster, J."

See: "Haskell vs. Ada vs. C++ vs. Awk vs. ... An Experiment in Software Prototyping Productivity; Hudak, P; Jones, M."

Lisp - A dynamically typed language had the lowest development time by a factor of 2.5x. Seems to be the same figure I'm giving, fancy that!

"He also misses the biggest add of typing your code which is that other people can understand it better."

Nope.

See: "How Do API Documentation and Static Typing Affect API Usability? Endrikat, S.; Hanenberg, S.; Robbes, Romain; Stefik, A."

You need documentation and whilst Java students may study with dynamic typing since they haven't been taught it, it's pretty clear that on it's own static typing provides almost no useful documentation.

"This is often lost on junior programmers..." I'm far more experience than you, which is why I'm calling out the BS.

I think you will find that the research suggests that static typing and dynamic typing have a similar bug rate per line of code.

See: "A Large Scale Study of Programming Languages and Code Quality in Github; Ray, B; Posnett, D; Filkov, V; Devanbu, P"

Once you combine that knowledge with the knowledge that dynamically typed programs have less lines per feature.

Relational Lisp 274 compared to Ada 767 or C++ 1105.

Dynamic typing wins out because the programs are smaller, making them faster to write and contain less bugs per software feature.

The link you posted confirms it.

Do you understand that you are wrong?


Any source for those made-up-sounding numbers?


Could you name some of those studies? I would be interested in reading them


We had an entire history of stupid errors (like field name typos), excessive test coverage (and associated maintenance burden) to catch those stupid errors, and everything breaking every time you upgraded a big framework (like Django) even with all of those tests in place.


Spoken like someone who's never had to step into a 2 million line python library and try to decipher or debug it.

Also, we've been using non-standard type hints for nearly all of python 2, they were just non-uniformly specified in docstrings and weren't easily referenced or introspected for tooling and IDE's to work with them.


And most of that python2 code is completely rotted.


By definition, since it won't run on a supported runtime.


All these features are nice and stuff... But they are often runtime features... At compile time Python doesnt tell you wether a program is correct. Which is fine for small programs or small services. But any big system is written in python is really hard to maintain without LOTS of unit tests...

Static typing, compile time checks just win in the long run. And with languages like kotlin you still have all the advantages, and you have nice tools like python has which execute at COMPILE time using DSL, which are type safe themselve..

Python has become mostly a toy language for me to write scripts in.


No programming language tells you if a program is correct at compile time. Type errors aren't a very common type of bug either.


The most common type error I see are unexpected null values. Which, surprise, is still a problem in many typed languages. :)


“No programming language tells you if a program is correct at compile time” is technically true, but my experience is that about 99% of the refactors I do in rust, no matter how large, go back to working correctly as soon as the code compiles again. I don't think anything like this is possible in a language like python.


That's true, but mostly because one rarely compiles Python.

I've done significant refactors in untyped Python without much ado: as soon as your tests pass, code works too. I've done significant changes with tests passing on the first go.


As I know, SPARK can do this (in particular, thanks to the strong type system).

https://learn.adacore.com/courses/intro-to-spark/chapters/01...


> At compile time Python doesnt tell you wether a program is correct.

This is a general limitation of type systems, IIUC. If you want a type system to guarantee 100% program correctness, I think it can't handle an arbitrary Turing-complete program.

Disclaimer: I'm not sure I really understand this topic, so take this with a grain of salt.


What does "compile time" even mean for Python?


The moment you run the script, before it actually gets executed.


Yes but a linter can


Before I read this article I was not aware of exclusive advantages to dynamic typing.

At least now I know what is achievable despite its presence.


Which of these do you think require dynamic typing?


A lot of the orm style things are not really feasible in static-er languages without clunky approaches like codegen.


What part specifically? Java is (in)famous for its orms after all. Unless you want to count that as clunky codegen? I'd argue that all metaprogramming boils down to that though.


Java is actually a very dynamic runtime with a very static language front end.


The thread is about programming semantics though, so what does that have to do with anything?


cough Reflection. This also includes Annotations which can give clear semantics to code generation.


Unless you want to count Java as clunky? Well, I definitely think Java is clunky. This article by Steve Yegge, titled "Execution in the Kingdom of Nouns" sums up Java for me: https://www.eecis.udel.edu/~decker/courses/280f07/paper/King...


... this PDF is from 2006, the same year that Java 6 was released. Java 7 and Java 8 were game-changers at their time.

I doubt there is much relevance left, especially in light of Java's excellent functional integration into its type system and JVM.

Edit: ``` There's no reason Java couldn't simply add first-class functions and finally enter the grown-up, non-skewed world that allows people to use verbs as part of their thought processes ```

Ironically, this is exactly what happened. With exception that proper technological implementation took many years and might have one of the best implementations for anonymous functions out of any programming language (thank you, dynamic JVM).


Don't forget the many additional features since then, such as records, pattern matching, virtual threads, and more.


Almost all Java ORMs are dynamically typed in the ways that matter. The "clunky codegen" method would be something like JOOQ, that actually checks your schema.


Java's ORMs are infamously clunky and both them and many of its modern frameworks make tremendous efforts to twist the language into an unidiomatic, dynamically typed thingy with a bunch of extra compilation in between.

By comparison using and modifying ORMs in languages like Python and Ruby feels like second nature.


that's the point, none of these examples are intrinsically linked to dynamic typing.


Goal directed-execution programming languages, such as 5th generation languages icon[0] & unicorn[1], would get the 100k (typed) / 10k(untyped) loc's down another factor of ten.

~1-2k loc around the range of being easy to get up to speed on what's going on than equivalent 100k/10k LOC program with less 'typing' issues. (compiler/interpeter & keyboard too!)

Haven't run across a uni-py language yet (python & unicorn mash up)

[0] : https://en.wikipedia.org/wiki/Icon_(programming_language)

[1] : https://en.wikipedia.org/wiki/Unicon_(programming_language)


While this might be about my most unpopular opinion, it feels like it is time to start putting together a Python 4.0. I don't particularly have skin in the game but there is(?) enough meat on the bone around things like improving the GIL status quo, JIT compiling, static typing, and presumably etc. to be worth a breaking change.


That's definitely an unpopular opinion. The 2 to 3 change caused a lot of pain. I can't see the python community attempting that again.


The GIL is already set to be optionally removable via a compile-time flag in an upcoming release. That's about as meaty a change as you can get, and it's still not going to require a 4.x release.


Maybe string can be ebcdic this time around?


His first example of python's Callable type-hinting not supporting keyword only arguments was true 5 years ago when type hinting was first catching on, but Protocols have existed since python 3.8 and they support keyword only arguments.


A fun piece of metaprogramming I did for a Django project was checking whether our models had overwritten str() methods: https://github.com/svthalia/concrexit/blob/4dc7691c2613b268d...


Don't fool yourself- if your programming language has an eval() function, you are writing in Lisp and you are a Lisp programmer now. Embrace it!

All joking aside, I do think Python is pretty handy.


Oh no, Lisp is so much more powerful. For example you can update the definition of a class in a running system and update all existing instances of that class. When an error occurs you can inspect local variables in any of the stack frames and execute commands in them. And that is only the tip of the iceberg, Lisp is, as far as I know, without equal in terms of power.


> you can update the definition of a class in a running system and update all existing instances of that class

Like this?

    >>> class MyClass:
    ...   attr = 0
    ... 
    >>> my_object = MyClass()
    >>> MyClass.attr = 42
    >>> my_object.attr
    42
    >>> MyClass.method = lambda self: self.attr
    >>> my_object.method()
    42

> When an error occurs you can inspect local variables in any of the stack frames and execute commands in them

What is the difference with https://lukeplant.me.uk/blog/posts/pythons-disappointing-sup... ?


To Lispers, Python's hyperpowers (reflection and dynamic typing) are just regular Lisp business. Lisp's strength lies exactly in how convenient doing metaprogramming is in Lisp.

There are probably some subtleties and differences in how powerful both systems are (can't say more, I don't know how powerful python's metaprogramming is), but just know that Lisp has practically no limit in how you can extend the language.


Adding/changing/removing slots & methods. But also adding/removing superclasses -> which causes changes in existing objects. Having functions called when an object changes, etc. That's old in CLOS, end 80s. No wonder if some other dynamic languages by now have similar features.


My personal experience is that static typing not only eliminates a huge number of unit tests but increases documentation expression in combinationn with good names.

Dynamic typing is excellent for experimentation though.

Python hits the sweet spot.

But I can definitely recommend Groovy as an excellent alternative having all the strengths of the Java ecosystem.

Scala and Kotlin are also quickly catching up.


Is there any language out there that improves on Python in various ways but still allows convenient access to literally the entirety of Python's ecosystem of libraries?


While I agree with the @elcritch reply (https://news.ycombinator.com/item?id=34616755) that Nim is just nicer overall, depending upon one's idea of "convenient access to literally the entirety", Cython may be more to your liking as mentioned else thread: https://news.ycombinator.com/item?id=34616052

While usually pitched as "only" a language for fast extension modules and/or bindings to existing C/C++ code, Cython is really a language in its own right. https://cython.org/ has some more details.

It works by generating C code that calls the regular CPython API, managing all the ref count and tuple jazz that's rather a hassle to write in C directly. Then you run a C compiler against this code and it literally just links against the CPython .so's/DLLs. So, it's not just calling module code the same way as CPython, but actually calling the core built-in to Python code the same way as well.

As you add more `cdef` type annotations to your cython code, the generated C code becomes closer & closer to hand-written C code (both in hazards and efficiency).

Of course, a caveat is that may be Cython also doesn't improve upon Python very much as a language, except for allowing gradual static typing.


I've come to really enjoy programming in Nim. Note that Nim is very different language despite sharing a similar syntax. However, I feel it keeps a lot of the "feel" of Python 2 days of being a fairly simple neat language but that lets you do things at compile time (like compile time duck typing).

There's a good Python -> Nim bridge: https://github.com/yglukhov/nimpy



F# is a great language... I wish I could use it at work.


One of the selling problems is that it can be hard to bring coworkers up to speed on it. Management don't like that kind of thing.

What management like very much is that using F# attracts better talent, but they no can believe.


It's actually very easy. If developers can't learn a slightly different language then they're getting paid too much.


F# is a highly different language (at least from Python and Typescript) with additional concepts and I don't think you appreciate the problem for most companies to hire developers at all, let alone the kind of people who can hit a new language without a speed bump.


I would think any developer worth his salary can pick up a new language in a couple of months at most.


PythonCall [1] and PyCall [2] provide pretty convenient ways to interoperate with Python from the Julia language.

It has great package/environment management, excellent (potentially close-to-C) performance, automatic type inference, and a good set of interactive tools that make for a rich REPL.

But any such FFI (from any language) is an additional point-of-failure, an extra gear that could break and has to be maintained. And on the con side of Julia, there's an initial compilation time (which is improving version to version, but still a factor to consider).

Also it should be mentioned that the advantage of Python is not only its ecosystem of libraries, but also the vast array of tutorials and learning resources, and of development tools. Those are areas where most other languages have to play catch-up with Python, especially a relative newcomer like Julia.

[1] https://github.com/cjdoris/PythonCall.jl [2] https://github.com/JuliaPy/PyCall.jl/


Hy: https://docs.hylang.org/en/stable/

I tend to stick to vanilla python though, mainly because Hy is too much of an hassle for my use cases.



GraalVM myb?


I maintain Python code bases for a living, and feel that the language has simply been pushed too far. Static typing in Python doesn't give you the advantage of actually static typing and even IDE support is -- well it's not terrible, just not great.

The thing is, once we go through all this static typing exercise in Python, we get no performance advantages, and the whole thing seems bolted on, with worse semantics than most modern typed languages. And yes, it can stifle undeniable advantages a dynamic language like Python can provide.

Python remains a fantastic prototyping and scripting language, but my feeling is that it now handles more than it should.


I routinely use cython to compile python for heavy workloads. A big part of the ~10x speedup i usually see comes from strategically assigning true static types to certain variables that get frequently iterated or compared. Most variables are left as standard python as it isn't necessary to change them for performance.

My experience has been that cython "just works" even with lots of external libraries etc, and the code ends up performing as well as any c++ code i could realistically write while being much more understandable.


> I routinely use cython to compile python for heavy workloads.

an alternative is rust + pyo3 https://pyo3.rs

here's a web framework written with it, https://robyn.tech

the other poster child for pyo3 is polars, https://www.pola.rs

it's simply, amazing.


I'm so glad I found Polars. I replaced the most time-consuming parts of my Pandas code with Polars and it has reduced data manipulation times by literally 90-95%.

Amazing is not an overstatement.


which of these links should I start with with zero xp?


If you don't know Rust, but know Python, you can install Python libraries written in Rust with pip. Like, pip install polars or pip install robyn. In this case you follow the two bottom links. But then you don't write your own libraries and stuff so.. I guess that's not what you want.

If creating your own Python libraries in Rust is what you want, you would check out the first link I sent only, the one for pyo3.

But, if you want to learn Rust, you probably wouldn't start out with pyo3. You first install Rust with https://rustup.rs/ and then check out the official book, and the book rust by example, that you can find here https://www.rust-lang.org/learn - and maybe write some code on the Rust playground https://play.rust-lang.org/ - then, you use pyo3 to build Python libraries in Rust, and then use maturin https://www.maturin.rs/ to build and publish them to Pypi.

But if you still prefer to begin with Rust by writing Python libraries (it's a valid strategy if you are very comfortable with working with multiple stacks), the Maturin link has a tutorial that setups a program that is half written in python, half written in Rust, https://www.maturin.rs/tutorial.html (well the pyo3 link I sent also has one too. You should refer to the documentation of both, because you will use the two together)

After learning Rust and building some stuff with pyo3, the next step is looking for libraries that you could leverage to make Python programs ultra fast. Here https://github.com/rayon-rs/rayon is an obvious choice, see some examples from the Rust cookbook https://rust-lang-nursery.github.io/rust-cookbook/concurrenc... - when you create a parallel iterator, it will distribute the processing to many threads (by default, one per core). The rust cookbook, by the way, is a nice reference to see some of the most used crates (Rust libraries) in the Rust ecosystem.

If you are doing async stuff in Python, you probably need to look up https://github.com/awestlake87/pyo3-asyncio and https://tokio.rs/ - that's two libraries that Robyn uses (see Robyn dependencies here https://github.com/sansyrox/robyn/blob/main/Cargo.toml)

Anyway there are some posts about pyo3 on the web, like this blog post https://boring-guy.sh/posts/river-rust/ (note: it uses an outdated version of pyo3, and doesn't seem to use maturin which is a newer tool). This post was written by the developers of https://github.com/online-ml/river - another Python library written in Rust


I think the issue with Python is that it is relatively underfunded compared to other languages of its size. The entire Python ecosystem lacks leadership. Packaging is a clusterfuck, Pypa today still doesn't contain package metadata (which massively slows down downstream resolvers like poetry and wastes terabytes of bandwidth every year). Non-security issues take forever to resolve. Nothing encourages parties to contribute, unlike Go or Rust. You end up with companies having their own subtly-imcompatible internal forks. The benevolent dictator never cared about performance until recently. Most of Python's changes have been simple syntactic sugar. Python as a compiler engineering project is an exercise in mediocrity compared to V8 or LuaJIT.


I think the mistake is to assume that Python's type hints form a complete static typing system. Of course they fall short if you think that's what they're supposed to be. I think of it instead as a sometimes-useful subset of checks that you might get from a static typing system, and I use it in those places in my projects where I think it will be the most useful. The very fact that I can think of it as an option I can enable puts it in a totally different space from traditional static typing systems.


Yea this is exactly how it feels to me. Like “reinforced dynamic”.


There have been some efforts to utilize the type hints to give performance boosts. There's a project called mypyc that apparently has been used by black (python formatting library) that will compile type hinted python into c extensions. Unfortunately I think development has stalled, but as more people start using type hints I think there will be more motivation for similar projects.


> Python remains a fantastic prototyping and scripting language

I think it sucks for prototyping too if your prototype is constantly evolving. I have a WIP with about 100 files in it that I've been aggressively evolving over the past year, and I already want to rewrite everything in something statically typed, because any kind of change became pain in the ass. Too bad the project requires Python-first libraries.


Large Python projects are done with microservices typically.


1. I would not call 100 short files 1/3 of which are simple tests "large". I've handled 10 times larger C# projects without any hiccups.

2. This is a deep learning research project. Files are layers, pure functions, and model builders. Can't really turn it into microservices.


That's also true of Typescript, which has been wildly successful and taken over the javascript ecosystem. The main advantage of types is that it makes your code more maintainable by adding guardrails. You'll still possible run into issues at runtime and they're not perfect, but they're better than not having types at all.


Yeah the difference is that Typescript has a single good implementation (Python has two and the worst one is the most popular), and people actually use Typescript, so your dependencies have types.

I've also found that Python developers tend to write highly dynamic code that is difficult to statically type, whereas JavaScript developers - even ones that don't use Typescript - have mostly realised that that is a bad idea (with some exceptions cough Vue).


Typescript really rarely feels bolted on to JavaScript. That also has to do with typescript introducing new syntax to JS.


There's very little new syntax added by TS


Honestly, static typing encouraging data shape documentation alone makes it worth it to me. Yes, they're pretty hacky in Python and don't feel great, but I'm sure it'll continue to improve.

I'm just so over having to guess what a function might accept or return. Life is too short to spend it constantly reverse engineering code because people can't be arsed to write proper documentation.


This would sum up my sentiment as well, just for documentation reasons alone it is wildly beneficial.

People who love dynamic typing seem to think their code is just “understandable” and it almost always isn’t


The problem with separate prototyping languages is that the most permanent fix is always a temporary one. The prototype usually becomes the final product. If your language isn't suitable for use as a final product, then you're going to have a problem. If it's not suitable for writing a prototype, then you already have a problem.


That's why I love the idea of a language like Common Lisp. Performance, 'debugability', and flexibility in one package.


I also maintain a large python codebase at work, typing makes things simpler. For example using pydantic 90% of serialization code just goes away. Mypy eliminates a large number of bugs. Also when working with multiple people in the same codebase typing makes intent so much clearer. The type system itself is not perfect but it's a clear improvement of not having it at all.


It's a net benefit. I just think that we've stretched Python into all kinds of directions which should be served by actually statically typed languages.


I could not agree more. I’m fairly language agnostic but typed Python with pydantic is a joy to work with.


I disagree, it makes things more complicated by bringing in concepts such as generics.

Using MyPy will on average double your bug count per software feature. You might not think it does that. But if you actually go and measure it, the bugs go through the roof. There are studies on this. It's for subtle reasons revolving around where bugs come from in code.


> the whole thing seems bolted on, with worse semantics than most modern typed languages

One of the biggest sources of ugliness is the "None". The standard way of declaring variables ahead of time is setting them to None. But then, the type hints just become these ugly unreadable Optional[ActualTypeofVar] everywhere.


> The standard way of declaring variables ahead of time is setting them to None.

I'm not sure this method is actually "standard". For many use cases there is an obvious "null" value of the desired type (for example, just set an int variable to 0), so there's no need to use None.

For use cases where you can't just initialize the variable to the "null" value of its type (because that value has some other meaning for your program), then the true type of the variable isn't just the type you're thinking (e.g., not just an int), because you need to be able to distinguish the variable having the "null" value from it not being set at all (the None case). And for that kind of use case, Python's type hints are correctly forcing you to declare that variable's type the way that reflects the actual situation.


You can declare a type-hint beforehand without setting the value of the variable. See: https://peps.python.org/pep-0526/#global-and-local-variable-...


Yes, but then those variables are not visible when we dir(object), which makes it harder for the user to prototype. Certainly, type hinting should not get in the way of the users.


That's because your type is Optional[TypeOfVar] though; if you try using that variable in a place that expects just TypeOfVar before it's set, you should get an error.


Let me restate. To prevent ugliness in typing, either

* Python should have a way of declaring variables without setting them to None, so we can type hint them to their actual types. Pre-declaration is very important for a prototyping language, and needs to be done in a way that the variable is visible to dir().

* Come up with a cleaner syntax than the wordy Optional[]. Thankfully, they have recently moved on from the ugly Union[a,b] to the more readable a | b. Something more readable could be done for Optional.


but Optional[foo] is literally a union of foo | None already!


    a: int

    b: str | None = None


Right, in langs with c-style scoping, this is an issue, but in python

    if foo:
        x = a
    else:
        x = b
Works, so there's no need to predeclare.


A very common pattern in python is to declare all instance variables of a class in the __init__ method by setting them to None. This is done for

* code readability; all relevant variables are in one place

* Python is a prototyping language, and a very common usage pattern is doing dir(object) to determine the names of all variables that could be set to something.


This is no longer needed with modern python, you can use

    class Foo:
        my_int: int

To handle this. That said, for the second case, a variable that may be set to something but may not be is optional! That's correct behavior!


Im used to just doing

  x: list[str]

  if foo:
    x = []
And if a variable is nullable, I use a union:

  x: list[str] | None = None
That’s much more readable to me!


Rust and Typescript are some of the most well-loved languages, and the typing semantics of python are almost identical.

Have you used VSCode? or PyCharm? Their typehinting support is pretty great. At least as good as anything for typescript.


I used to be a huge dynamic languages fan, with python being my favorite language over the majority of my career. Then I worked on a large python project of ~100k LOC with a team of ten. That's when I realized that writing code faster isn't the problem. Reading it is, making changes to code someone else wrote is, and refactoring across dozens of modules is a problem. Static languages help a lot with all three. I still love dynamic languages for small tasks, but I'd rather use a static language like Go which still keeps much of the dynamic feel, and helps catch my mistakes at compile time. I actually just set it to watch the project directory for changes and recompile automatically in a terminal on another screen (actually to run the unit tests, which includes compiling.) That's constant feedback and comparable to dynamic languages speed for edit-compile-test runs. But the best part is the unit tests. They run so much faster with Go that I don't get distracted waiting for them to finish, and that keeps me in the zone and much more productive.


This is almost always framed in a way that shows large systems need the extra rigidity of some static systems. And that dynamic systems are only about rapid prototyping.

I think that is a bit of a false framing. To wit, any system of 100k LOC will be hard to get into. Even harder to make changes in. There is no getting around that. None. Even worse if you have many entities flying about these 100k LOC. Each change to an entity will be dangerous. Static or dynamic. Especially if they are persisted anywhere, as at that point it is all too easy to get static guarantees that aren't reflecting actual data.


This is a false equivalence.

You need all the help you can get maintaining large codebases and static typing is a huge help.


What did I make an equivalence between? Large codebases are hard, period. That is all I am saying. Having worked on 100k LOC systems in static and in dynamic, I will not claim either has a benefit. That is just hard.

Static analysis is, of course, good. If that is static typing or otherwise. So is running the code making sure it does what you want when running.

What is not a huge help, is a byzantine type hierarchy that nobody understands. Which is all too familiar to me from any codebase I have worked on with folks that dived straight into the type system before they knew what they were doing in the system.

Many strong type proponents have moved to the categories view. If you know the category type of what you are doing, the idea is that that will prevent bugs. But... that hasn't been my experience. Often it just hides what is actually happening behind another layer of jargon. Jargon that is not native to the problem being solved.


"I will not claim either has benefit"

I bet that if you did a survey of developers who have worked on projects of this scale, 90% would disagree with your opinion.


Then do said survey? Find empirical evidence and share it.

My experience, sadly, is the louder a dev on the team is about either dynamic or static typing, the more likely that dev's code is something nobody else in the team wants to work with.


Here is one sample point for your survey.

Static is better than dynamic for large projects, in my experience.


You should have more than one point of data. How many software systems? How old were they? How many contributors? How many bugs in each? Were any of them green field?


Dude, you have already made up your mind, so I am not going to do any homework for you that you yourself aren't performing. You are not my boss, or anyone's boss here


Amusingly, you don't know my actual stance. Which is probably closer to pro static types than to dynamic.

My criticisms in this thread is that the static type brigade does not hinge on evidence. It is typically hollow claims and getting angry at dynamic languages for being obviously bad for lack of helping.


Hard evidence is hard to obtain for something as varied and unrigorous as software.

After 25 years of working on all kinds of codebases I’ll take even badly engineered statically typed code over dynamic any day of the week including Sunday.


I wouldn't.

Badly designed software with terrible class structure (FooModel, FooSchema, FooSpec, FooSchemaSpec, FooSchemaSingle, FooSpecContainer with inexplicable relations between them) is way worse than a well designed non-explicitly-typed software.


I don't know, I've worked on a C++ project where most of the Ids/Index 'types' are done with typedef/using int :-(

So in theory the language is statically typed but in practice case a large part of the project is 'integer typed' ..


You are not going to find any kind of hard evidence for anything when it comes to software engineering.

If you have some experience in software engineering you should already know that.

Based on the "hard evidence" in this thread you are either ignorant or a time wasting pedant.


+1


The equivalence you're making is that dynamically-typed large code bases are just as difficult to get into as statically-typed large code bases.


This implies that "difficulty" is a totally ordered set. Which... I find highly unlikely. Things can be very difficult in amazingly unique ways.


Large codebases tend to be harder than smaller ones within one language. But language to language, static typing is such a significant help that it makes a large difference.

I’ve been writing in python for 18 years, and yet I can far more quickly get into a large typescript codebase than a python one of equivalent size.


Amusingly, I hate python.

Static typing is a tool I like. I don't think we have evidence that it is an obvious win. And I've bounced off of several typescript code bases hard. To the point that I currently hate typescript. Despite being impressed with some of its capabilities.

Which is again to say ymmv. Bad code is bad with or without static types.


> any system of 100k LOC will be hard to get into

To get a gut feeling of the system, i.e., to understand the 10000 foot view of a 100k LOC system, is orders of magnitude easier with statically typed languages than dynamic languages.

And that is mostly because those 100k LOC are done by probably dozens of developers and that leads to different styles, approaches, conventions, etc.

A statically typed language would remove a lot of those headaches, because it does force a semblance of structure, that can be easy to reason with.

A language like Python is awesome for a micro-services architecture though.


I call bs in this. To get a 10000 foot view of a codebase, you lean on diagrams and other lies of documentation. Note that I don't mean malicious lies. I mean simplifications and other happy path discussions of how things work.

This is no different from any system. Want to know how your car works? Start with a simplified diagram and gradually add more details.

Cars and other equipment are an amusing case study. What is the strongly typed version of a car schematics? Why does it look so different from what we think the idealized software should look like?


Anecdotal but — I've worked on many-million-LOC codebases in Ruby with large teams (pre-Sorbet and other type checkers, at Airbnb), and many-million-LOC codebases in typechecker-checked Python with large teams (Instagram). The typechecker-checked Python was way easier to reason about and refactor.


Dynamic languages large code based are, in my 14yo experience, much much harder. Today no one a serious project with javascritpt, AFAIK everybody use typescript. Why?


JavaScript is a badly designed language. It was created on the fly over several decades by browser makers. People prefer to code in absolutely anything else.

WASM will probably kill TypeScript shortly.


Actually it was written (so it was done) in roughly two weeks and then released. The rest is part of modern history in terms of distribution of software and backward compatibility/portability.

The two other languages you mention, TypeScript and WASM, are more part of the same platform mutually benefiting from the progress than competing each other.


Actual JS is based on original one, but have been much improved. Still the lack of static typing is a liability fixed by typescript.


A ~100k LOC project in a statically typed system with type hierarchies, interfaces, contracts and boiler plate will boil down to ~10k LOC in a dynamically typed language like Python. A 10k LOC project will be more readable than a 100k LOC project.

Source: I have spent years coding in C++/Java, then Python. I have migrated Java projects into Python


10x seems extreme. ESR went from 14k lines in Python to 21k in Go. http://www.catb.org/~esr/reposurgeon/GoNotes.html


I didn't mean to exaggerate. I think part of the improvement was the luxury of refactoring which should generally reduce the bloat. And, as someone else said, part of the issue is C++/Java, not static typing. When I move from Java to Python, I also get the luxury of organizing code into fewer/meaningful source files. I find this more readable, than having to switch to different files constantly.

I have not had a chance to learn or use a language like Go. But production use of Python, including building large code bases is real. We do resort to numba, cython or using Python API to compiled code.

I'm now involved in converting large codebases from SAS to Python. I don't think I will have the luxury of choosing another language like Go, for a number of reasons.


It's funny, both Java and Python emerged in the 1990s, yet Java feels incredibly crusty and awkward in 2023. I wouldn't choose Java as a fair representative of statically typed languages in 2023, but Python still feels like a reasonable choice for a non-Lisp dynamic language.


I'm wondering how many of the extra lines are those containing only closing bracket


You probably were replacing LoC with libraries or syntactic sugar, you can do the same with java. As a senior I can write same functionality than juniors with half LoC. In my experience Python is totally unusable when dealing with poorly documented 3rd party libs.


Your problem is C++/Java, not static typing. Static typing does not add many extra lines of code. In most cases, it adds no extra lines of code, as declaring variable/param types is done inline.

Heck, just look at static typing in Python.


Static typing is not only about declaring your arguments and variables as simple types (int, float, array of ints etc).

It means having a distinct type for any complex structure you pass around (think pre-normalization API params, post-normalization API params, slightly enriched post-processing data as separate type vs a dict of str to anything in Python), or anything you want to make compatible (think interfaces vs duck typing).


So Generics, Traits, Abstract subclasses, etc... don't exist in static typing?

As soon as the types get complicated, the code bloat begins.


TypeScript is the language Python wants to be. If only JS came with a Go-level stdlib there’d be nothing to complain about.


Seems valid, most modern Python code is almost the same as Typescript without {}


Python the language doesn't give you super powers and I would say that its clunkier and less elegant than Ruby and obviously not as powerful as languages like F#. Python's ecosystem and contributors are its biggest benefits and they help you succeed in spite of the language itself.


The ecosystem exists because Python does make it possible. That was one of the main thrusts of the article!


> The ecosystem exists because Python does make it possible.

Only to a certain point, after which the network effects override everything else.


Let's stop calling "dynamic" and "static" versus "runtime" and "comptime" instead ?

Also, "dynamic code generation" for "meta programming" ?


static and dynamic refers to where you find an object, not when it lives.

Some people like to know which type of meta programming is being used at a given point. This is something akin to "all dynamic code generation is meta programming, but not all meta programming is dynamic code generation".

One should be careful to not merge or eliminate words with subtle differences. They usually exist for a reason.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: