FWIW, I wrote isort, but am seriously considering migrating my projects to use Ruff. Long term I think the design is just better over the variety of tools we use within the Python ecosystem today. The fact we have a plethora of projects that are meant to run per a commit with each one reparsing the AST independently, and often using a different approach to do so, just feels untenable long term to me.
Yes, as of last month. I’m not sure if it works with YAPF; it’s designed to work with Black and doesn’t currently have many of isort’s configuration options. Worth a try!
Yes. The code started as being indented by tabs, so changing it now is a mess. Also, not to start a flamewar, but I've always preferred tabs to spaces in any language, I find them more reliable, easier to use...
Not going to argue about your personal preference, but in Python spaces for indentation isn't just a personal preference - you'll run into a fair amount of issues with that, this one being just the tip of the iceberg.
Apart from isort not knowing how to deal with tabs, and changing the defaults "spaces" to "tabs" in the tools (along with line length too, because for some reason the 80 character limit is set in stone even though it's absolutely outdated) I haven't had many issues. And this has been running for years already.
The key to success with tabs in any language is simple: Don't use alignment.
Instead, only use indentation.
It is interesting to note that Black-formatted code uses indentation only and never uses alignment. It would be perfectly compatible with tabs, unlike Google-formatted code which relies heavily on column alignment.
If your import statements are indented, they must be in control statements (try/except or conditional imports, I fail to see why you would put those in a loop) thus they will be difficult to reorder if you import another module (with different name or stdlib status) on import error.
> If your project builds a Docker container, also create a .dockerignore file to specify files and directories that should be excluded from the container.
I would nitpick this. You build images not containers and since files are not copied by default there is more nuance here that the .dockerignore file makes builds faster by not including them in the build context.
That does ultimately prevent COPY directives from using them but it is these sorts of brief, slightly inaccurate summaries that mislead folks as they build understanding.
Shouldn't the speeding up of the build make the program less boring?
From my understanding, the program gets more boring as the time it takes an application to build increases.
> slightly inaccurate
Not entirely, I'm not sure the author even wanted to stress on this in the article. People won't learn docker from a python article about the same.
Not sure if I like the recommendation to not let Black change your code and just give out errors.
I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code.
Black shouldn't break anything (and hasn't broken anything for me in the years I used it) but in the unlikely case it does it, there's still pytests/unittests after that that should catch problems...
As I understood it, it was to not let black do the formatting during CI builds. In local dev you’d let it reformat.
Even while it won’t break anything you want CI to be your safety net, flagging a local setup as being wrong is more valuable than magically autocorrecting it.
> CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such.
That going too far unless you define code to be a subset of the files checked into the repository and simply define any file that's touched in an automated manner to be not code
There are a lot of useful automations that can be part of the CI/CD pipeline, such as increasing a version number, generating a changelog, creating new deployment configuration etc
They don't have to be part of it and it's possible to work around it/don't commit... But that comes with it's own challenges and issues
It's not about not trusting automated generation of certain artifacts. Stamping version numbers may be a fair idea. (A changelog entry needs human review and approval, in my book.)
It's mostly about the flow of data and control: source files, beside some known auto-generated files / single lines, are the source, and whatever is generated is downstream from them, not altering them. It's like a React app: data flows through props in only one direction, you don't patch DOM in event handlers, or something.
Its completely fine if you wish to treat your repository like that, but that doesn't make this treatment the only viable and correct way.
There are a lot of people which include their helm package as part of their project repository and even more that generate the changelog from standardized git messages like conventional commits and still wish to persist them in a changelog.md inside of the repository to make completely banal examples spelled out to the letter. It works well and lets you scale these things pretty well inside of a corporation with a lot of teams.
Its completely fine to keep all that out of your automated pipeline, triggering only after these things have occured... but that's not the only viable choice a developer can make and you're very much talking from of an extremely limited perspective if thats your point of view.
This is the way. We have format-on-save in our editor, works like a charm. Sometimes the CI still catches sometimes, but generally its very low friction.
My current project is my first project in a while which does not use black.
I liked black, though I was never satisfied with the fact that there was no way to normalize quotes to be single quotes: '. Shift keys are hard on your hands, so avoiding " makes a lot of sense to me. But there's the -S option that simply doesn't normalize quotes so it has never been a real issue.
However, this new project has a lot of typer functions with fairly long parameter lists (which correspond to command line arguments so they can't be broken up).
black reformats these into these weird blocks of uneven code that are very hard to read, particularly if you have comments.
Everyone is a fan of black; no one liked the result. :-/
I have a key in my editor to blacken individual files, but we don't have it as part of our CI. Perhaps next project again.
> I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code
100% this. I also let Black auto-format code in the CI and commit these formats.
A lot of developers, intentionally or not, don't have commit hooks properly setup. If Black doesn't change the code in CI they need to spend another cycle manually fixing the issues that Black could have just fixed for them.
You're saying that there's a risk that Black could break your code when formatting? Well, so could developers and I'd trust a machine to be less error-prone.
I don't understand why people are against this so much. Black does a sanity check and compares the AST before and after to make sure there aren't any meaningful changes (unless you are running it with --fast). So there is almost no risk that it will break your code.
There is nothing more frustrating than coming back from a coffee break only to find out that you have to rerun your CI check because of a trivial formatting issue.
If you use feature branches, it is quite annoying that can't push two consecutive commits because CI changed something in your branch and now you have to resolve conflicts.
We have it set up to only run after you've opened a PR. You shouldn't run into the issue often that way because if you're opening a PR then all your big immediate changes are already committed so you won't be pushing another commit until someone reviews which will be after black has already finished.
Sometimes I still sneak in minor changes after I opened a PR. Sometimes I open PR early because CI has integration/e2e tests that are hard to run locally. Sometimes I want feedback on certain parts early on and PR is the easiest way to show something.
There can be workflows in which pushing through CI works fine, but as a general advice it's not great because there are many edge cases.
> Not sure if I like the recommendation to not let Black change your code and just give out errors.
Let black format code before it is checked in. Code should not be reformatted for CI or production, and bad formatting should either ALWAYS throw errors (no known defects allowed) or NEVER throw errors (if it passes tests & runs ship it). Consistency is the key.
Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific -- this seems like a straw man argument against typing that doesn't acknowledge python's own advice.
Also, mypy has gotten really good in recent years and I can vouch that on projects that have typing I catch bugs much much sooner. Previously I would only catch bugs when unit testing, now they are much more commonly type errors.
The other thing typing does is allow for refactoring code. If anything, high code quality relates to the ability to refactor code confidently and typing helps this. Therefore I would put it at the top of the list above all the tooling presented (exception I agree with ci/cd)
Iterable is an import away, while list is already at my fingers.
There's zero harm in using list in private interfaces: I know I'm the only one passing the value, I know it is always a list.
As an argument type, Iterable is compatible with list, so it's benefits are minimal (with rare exceptions).
Lists are easier to inspect in a debugging session.
Iterable can be useful as return type, because it limits the interface.
Iterable is useful if you are actually making use of generators because of memory implications, but in this case you already know to use it, because your interfaces are incompatible with lists.
I can count on fingers of my hands when using Iterable instead of list actually made a difference.
> As an argument type, Iterable is compatible with list, so it’s benefits are minimal (with rare exceptions).
Iterable is not compatible with list, but list is compatible with iterable. As the more general type, Iterable is better as an argument type unless you have a reason to force consumers to use lists. Even in private interfaces, I tend to prefer it, because I often end up wanting to pass something constructed on the fly, and creating an extra list for that rather than using a genexp just seems wasteful.
What I meant is argument marked as Iterable is compatible with list being passed.
> Iterable is better as an argument type unless you have a reason to force consumers to use lists
See, I feel the exact opposite: I use Iterable only if I have a reason to force consumers to use Iterable.
When you're marking argument as Iterable, how confident do you feel that you will never query collection size or access it by index?
I understand the desire to limit the interface and YAGNI, but since lists are more familiar and ubiquitous, using Iterable feels more complicated and unnecessarily pedantic.
> See, I feel the exact opposite: I use Iterable only if I have a reason to force consumers to use Iterable.
A broad argument type doesn’t force consumers not to use a narrower type. (It forces the implementer of the function to not rely on additional features of the narrower type, but if I am writing the function, I can be certain whether or not that is acceptable.)
Meanwhile, using a narrower type than needed for an argument does impose additional, unnecessary constraints on the consumer.
> When you're marking argument as Iterable, how confident do you feel that you will never query collection size or access it by index?
Absolute certainty, since I know what the function does and what I need to do it.
> I understand the desire to limit the interface and YAGNI, but since lists are more familiar and ubiquitous, using Iterable feels more complicated and unnecessarily pedantic.
Since all lists are Iterables but not all Iterables are lists, Iterables are necessarily more ubiquitous than lists.
> Since all lists are Iterables but not all Iterables are lists, Iterables are necessarily more ubiquitous than lists.
Yeah, that's what I meant by being pedantic :)
Here's a question: you receive a JSON payload that contains a list. You will then pass this list to two functions, one of them only iterates, another one uses list interface (let's say checks length among other things). Should you mark the argument as a list, or as an Iterable in the first function?
Solely from the code perspective, it's definitely an Iterable. But in my mental model it still remains a list. I don't like it when code deviates from my mental model. Forcibly treating it as an Iterable only makes it more complicated, while not giving anything in return.
Sure, you could say that callee should not have expectations of the caller, but what if those functions are already coupled? They are in the same module, and argument names clearly denote a collection. The fact that in certain scenarios it is "technically Iterable" serves nothing but pedantic value.
> Solely from the code perspective, it's definitely an Iterable. But in my mental model it still remains a list.
A list is an iterable with special additional features, so this is no conflict at all.
> I don't like it when code deviates from my mental model
But how is there a deviation; being an Iterable is part of being a list, not a deviation from it.
> Forcibly treating it as an Iterable only makes it more complicated, while not giving anything in return.
How is there anything “forcible”. Broader typing doesn’t “forcibly” impose anything. And it does give something, more freedom to callers.
> Sure, you could say that callee should not have expectations of the caller, but what if those functions are already coupled?
If there is coupling that exists for good cause and demands a list as the type of the data structure to be passes around, then, fine, use list. But usually Iterable or Sequence makes more sense; coding to interfaces which impose only what is actually required is better than to unnecessarily specific concrete types.
> But how is there a deviation; being an Iterable is part of being a list, not a deviation from it.
Because I'm a human, not a robot. If I can describe something with fewer words by sacrificing a little bit of accuracy, I might go for it when I don't deem that accuracy to be important.
If wife sends you to buy eggs, are you the type of person that buys caviar because "technically they're eggs"?
> coding to interfaces which impose only what is actually required is better than to unnecessarily specific concrete types
> The other thing typing does is allow for refactoring code.
No. What allows you confident refactoring code are automated tests. I honestly can't understand why people are so obsessed about types, especially in languages like Python or Javascript.
It's not just about types. It's about having interfaces I should expand on. And, I'm assuming there are automated tests, otherwise and typing is additive. I should clarify that it's also having defined interfaces using the type system to do it.
By depending on interfaces/abstractions instead of specific cases you can refactor the interface and not break clients. It's very difficult to do this unless you have types.
This is something that Go is really good at and encourages but can be done with python/js on top of their type systems.
> I honestly can't understand why people are so obsessed about types,
Types in Python feel like an added layer of confidence that my code is structured the way I expect it to be. PyCharm frequently catches incorrect argument types and other mistakes I've made while coding that would likely result in more time spent debugging. If you don't use any tools that leverage types you won't see any benefit.
> I honestly can't understand why people are so obsessed about types
It's a very powerful sanity check that lets me write correct code faster, avoiding stupid bugs that the unit tests will also, eventually, find.
And, to me, reading the code is much much nicer. Types provide additional context to what's going on, at first glance, so I don't have to try to guess what something is, based on its name:
> I don't have to try to guess what something is, based on its name
It's probably just a bad example, but in case it isn't:
Sounds like you ended up at the same place. You went from guessing what is some_api.get_results(), based on it's name, to guessing what is SomeAPIResult, also based on it's name.
If some_api is your library, then you could have just added type hints to get_results() and let type inference do it's job.
If it's a third party library, then using your custom SomeAPIResult means that code is becoming alien to other engineers that worked with that library in the past. It might be worth it, but it's definitely controversial. You probably should've done it with stubs anyway.
I disagree. It’s not a guess, it is precisely what it is, where the variable name is free to betray me. A sane IDE/linter will tell me if my local assumption is incorrect, where a variable called result_SomeAPIResult relies on an assumed, possibly ancient, state of reality.
Type inference is good for the writer, not the reader.
Relying on type inference isn’t some rule. Your can find many projects that use it selectively, being explicit where it makes sense. The point of writing code is to make it readable and maintainable. The explicit type isn’t redundant, it’s explicit in presentation, and can be functional, like my example.
I mean, just look at this example. You know the type without having to dig in, do you not? You don’t have to look at the function definition. You know immediately. That’s the point of being explicit, where it makes sense. No guessing, where it makes sense. This is why we have all these type hints now, in a dynamic language: because guessing sucks.
I got good use of the run-time type checking of typeguard [0] when I recently invoked it via its pytest plugin [2]. For all code visited in the test suite, you get a failing test whenever an actual type differs from an annotated type.
> Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific
These statements contradict themselves? List is too specific, and Sequence[item] is preferred. Sometimes you are dealing with a tuple, or a generator, and so it makes more sense to annotate that it is a generic iterable versus a concrete list.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Yet the Python type-annotation ecosystem was strongly oriented around nominal typing (i.e., caring that something is exactly a list) from the beginning.
I'm saying that this quote is a straw man and that contrary to what is claimed in the quote, instead, the ecosystem would go with/recommend Iterable[Item] or Sequence[Item] and not List[Item] if applicable.
I think we both agree, not sure which part of my comment you think is contradictory.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it.
This is an odd complaint. typing.Sequence[T] has been there since the first iteration of typing (3.5), for exactly that use case, along with many related collection types.
You should never be using static typing with a scripting language like Python or Ruby.
Dynamically typed code is 1/3rd the size of statically typed code, that means that one developer who is using dynamic typing is equivalent to 3 developers using statically typed code via MyPy.
Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
This is confirmed by all the studies that have been done on the topic.
If you use a static type checking with Python, you have increased your development time by 3 and your bug count by 3.
Static typing's advantage is that the code runs a lot faster but that's only true if the language itself is statically typed. So with Python you have just screwed up.
Ah no I meant a proper peer reviewed source. The claim that untyped code has fewer bugs is completely bonkers, so I was quite sure that no such source existed.
Why do you think microsoft, google and facebook are all in the business of typechecking python? If typechecking would actually introduce bugs, it'd be better not doing it right?
Using github for statistics is flawed. There are millions of 10 line js libraries. Yes it's easy to not make type mistakes in 10 lines. I suppose that type errors increase more than linearly with size.
> The claim that untyped code has fewer bugs is completely bonkers
Not really. It is, however, quite expensive to measure, because dynamic typing really shines at the evolution of software, that is being able to respond fast to changing requirements. Legos vs play-doh: https://weblog.jamisbuck.org/2008/11/9/legos-play-doh-and-pr...
> Why do you think microsoft, google and facebook are all in the business of typechecking
A billion flies can't be wrong? Companies with unlimited amount of money are not the right place to search for good practices. Both Facebook and Google became flush with cash way before modern type obsession. Sure, once you are a multi-billion dollar company slowing down can be a good thing. But you need to get there first.
> If typechecking would actually introduce bugs, it'd be better not doing it right?
If sugar caused us to die sooner, we'd be better to eating too much if it, right? And yet, here we are.
"The claim that untyped code has fewer bugs is completely bonkers"
There are plenty of academic sources that will tell you that the number of bugs in a program is directly proportional to the number of lines in the program and static typing has no effect on this.
Additionally, statically typed code involves large amounts of boilerplate code in the form of abstract base classes, interfaces, generics, templating, etc. It's a very verbose code style.
It's your turn, find an academic source to backup your claim that static typing reduces the number of bugs. Cause it just isn't true.
Microsoft, google and Facebook have a lot of programmers coming from languages with static typing and want to make Python more familiar.
It's a far distance away from anything resembling good practice.
Actual Python houses typically don't use static typing.
Given that type hints are a new language feature in Python, please explain to me how these Python houses used static typing when type hints didn't exist?
> Given that type hints are a new language feature in Python, please explain to me how these Python houses used static typing when type hints didn't exist?
They've existed since 2015… please explain me how 2015 is "new"?
I have experience in working on statically typed Python codebases, it's just obviously inferior.
I'll go further and tell you the most common reason for using static typing is to allow the codebase to be a monolith like it's still the 90s. You shouldn't be trying to build a monolith in a scripting language it's a recipe for disaster.
I'm a polyglot, I'm exactly the sort of person who should be commenting.
Does it surprise you that static typing is often a poor choice?
Static typing is something to be used when the performance of your code is important. I've done 40 Gbits/sec network traffic processing, certainly static typing is used for that.
Your standard business CRUD app? Usually dynamic is the better choice.
> I have experience in working on statically typed Python codebases, it's just obviously inferior.
Maybe they just were inferior projects?
I've used a library where every function just accepted "args, *kwargs" and no documentation was given. In that case it's not really the fault of the language that it sucks. It could be a similar case for you.
> I'm a polyglot, I'm exactly the sort of person who should be commenting.
Ok I speak 3 languages fluently and 1 more so-so… But what does this have to do with python typing????
In fact most people who study literature and languages don't know much about python types.
> Does it surprise you that static typing is often a poor choice?
No because it isn't true.
> Static typing is something to be used when the performance of your code is important.
We can all agree that python isn't something to use when performances are very important.
> Your standard business CRUD app? Usually dynamic is the better choice.
Ok. That's not what I do though.
Have you done any C and C++? You know how people prefer doing a list in C++ with a template rather than a list of void in C? Same thing in python. But perhaps you haven't experience in this field either?
That seems to completely forget the fact that libraries exist, that A LOT of bugs can happen calling libraries, and that you're not really supposed to unit test libraries, they are supposed to have their own tests.
Unit tests test the parts of the library that are in use implicitly.
As does integration testing, user acceptance testing and whole system testing (QA engineers, frontend testing, etc.)
Nothing has been forgotten. Python isn't Java, nor should you develop your Python code as if it were Java. Python has it's own software development practices that take advantage of the language's strengths including dynamic typing.
It you don't know how to work with a dynamically typed language properly, that's on you. And I guarantee you will get poor results pretending it's a statically typed language.
> It you don't know how to work with a dynamically typed language properly, that's on you. And I guarantee you will get poor results pretending it's a statically typed language.
Insulting me won't make you a better developer nor a better person :)
Getting started with type hints is not easy, but it can be done incrementally and it is worth it.
If you haven't used them, please learn before insulting people who are more experienced than you.
Using static typing in a dynamically typed language will always make you a poor software engineer who doesn't understand the tools they use nor the engineering trade-offs involved.
But sure continue to hammer in nails with a screwdriver, it's only your own time you are wasting.
And I've noticed you failed to provide any sources whatsoever for your nonsense.
> And I've noticed you failed to provide any sources whatsoever for your nonsense.
As opposed to your 404 "peer reviewed paper"? :D
> Using static typing in a dynamically typed language will always make you a poor software engineer
Making broad statements about things you've never used yourself just makes you an arrogant guy on the internet who is quite likely to be a poor software engineer with a big ego.
Your "sources" were a blog with opinions based on questionable statistics methods and a stackoverflow with a 404 link.
> My wages disagree with that statement. But hey, we can't all be 10x.
Just a hint: this screams insecurity.
Wealth does not correlate with knowledge.
Especially since you don't even use type hints, so you have no professional experience with them. Just an ill formed opinion by reading blogs and comments.
> Did it ever occur to you to think about why people use scripting languages or what advantages they have over regular programming languages?
No compilation time? Very complete standard library? Opt-in typing? Good introspection? Numpy?
> Of course not, you are used to statically typed languages and are blinded to the idea that there are other ways to develop software.
I was doing python long before type hints existed, and I assure you they are an advantage. Now, I understand you lack the expertise to realise that. I'm just saying that insulting me won't make you correct.
> I'm guessing you did a search for sources to backup what you are saying and found out pretty quickly that they don't exist.
I did not bother. The fact that microsoft, google and facebook invest money into it is proof enough. You reject it because you're being irrational.
> Static typing in Python is a practice based entirely in common ignorance not reality.
I wish I could sound so bold and certain when being wrong!
Yes everybody knows what a cargo cult is. It's not some sort of intellectual remark. More of a random thing to say on the internet when you have no real arguments.
But good to know you can find sources… when they happen to exist :)
No offense, but it's like a textbook on awful debates.
Your best arguments are: "big guys are doing it therefore it's good for me too" (argument from authority) and "I tried both and decided only one is good" (argument from authority/anecdotal evidence). Plus a ton of ad hominems.
Please see Raymond Hettinger's keynote on efficiently handling bugs[0]. He makes the case that static type checking is a boon for Python except for in specific programs that make extensive use of covariant and/or contravariant types.
> You should never be using static typing with a scripting language like Python or Ruby.
You should use it where it makes sense, and not where it doesn’t. I haven’t used any of Ruby’s type checkers, but Python makes this easy enough; make what has a reason to be dynamic dynamic, and have static safety rails everywhere else.
(This is true with many “statically typed” languages that have dynamic escape hatches, too, not just traditionally “scripting” languages.)
> Coverage measurements are too easy to “game” — you can get to 100% coverage without meaningfully testing all or even most of your code
Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.
Hard disagree: 100% coverage is not a "good low bar" and does not increase code quality.
Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits.
I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
You can have bad unittests which make the system worse and you would be better of without them.
You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all.
The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that.
The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts.
Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.
> I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
It's a red flag to blame high coverage for fragile tests. Use narrow public component interfaces to reach code parts and you simultaneously gain robust tests which can be used during refactoring and you can be guided by coverage to generate test cases. Bob Martin has a great article: https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...
One useful technique for checking whether the tests are actually meaningful is mutation testing - mutmut is a great Python implementation: https://mutmut.readthedocs.io
Absolutely not. This leads to testing being invasive and driving the design of your software, usually at the cost of something else (like readability). Testing is a tool, you can't let it turn into a goal.
Could you elaborate with python-oriented examples? I tend to agree for static typing languages like Java — to fully test you have to go a total DI path. It leads to bloat and additional layers. I don't see anything similar for python because you have to do nothing to bring your code into a test environment.
> Testing is a tool, you can't let it turn into a goal.
Yep, and I use testing as a tool to be sure we ship quality code. It's 2x important for our case, we don't have control on hosts where our product is run and 100% coverage was a salvation. We even start to ship new versions without any manual QA.
My examples are just personal anecdotes, you can dismiss them by saying "our team won't make the same mistake".
If your goal is 100% coverage then it will turn testing into ritual and only give you the illusion of quality. Instead of testing inputs and edge cases, you will focus on testing lines of code.
I don't understand. The title of the post is: "Boring Python: code quality". Further down: "Today I want to talk about what's generally called "code quality" - tools to help...". I'm sorry but "code quality" is not "tooling". The post should be titled: "Python tooling". Code quality: What abstractions are you using in your code?, How easy is to make a change?, How easy is to understand your code base?, What patterns are you using and why?, Are you abusing class inheritance?, How many side effects are present out there and how does that affect your program?, Are you taking advantage of the Python language facilities and idioms?, Is it easy to write unit tests for?, etc. To sum up: "tooling" != "code quality".
> This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.
It doesn't really matter if it is fun, sad, entertaining or boring Python. The post wrongly claims that putting all these tools in a project will lead to "code quality". It says that at the very beginning as I quoted it. This is harmful, especially for a junior developer or someone that doesn't have much or none experience coding. It will make the naive reader believe that having those tools in place quality code is being produced.
While it doesn't produce quality, it can help a developer write better code; I've learned tons about JS internals just by sticking to whatever eslint popped up with. Before that I learned tons about Java's internals and best practices by fixing issues that these automated "best practice" tools came up with.
It's far from perfect, but it helps if you don't know any better. And most people don't know any better.
I'm currently spinning up a new project (React Native, Typescript) and I'm spending a lot of effort in locking down the project - eslint, unit tests & coverage, CI, strict typescript rules, etc - because this did not happen with the previous iteration of this project, leading to tens of thousands of LOC worth of unit- and end-to-end integration tests to become worthless and unusable. Sure, that was a lack of developer discipline as well, but why rely on other people when you can do it through technology as well? You can't control everyone.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it.
Terrible advice not to use type hints and this reason makes no sense. There's already pretty good support for Sequence and Iterable and so on, and if you run into a place where you really can't write down the types (e.g. kwargs, which a lot of Python programmers abuse), then you can use Any.
Blows my mind how allergic Python programmers are to static typing despite the huge and obvious benefits.
It's true that Python's static typing does suck balls compared to most languages, but they're still a gazillion times better than nothing, and most of the reason they suck so much is that so many Python developers don't use them!
> I recommend using two tools together: Black and isort.
Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before.
All those big changes introduce commits that make git bisect generally slower. Which might be awful if you also have some C code to recompile at every step of bisecting.
Set black up in the pre-commit with a specific version. When you make a commit it will black the files being committed using the specific version of black. As it's a subset, it's fast. As it's a specific version, it's not going back and forth.
It doesn't… people use a million different distributions. Forcing everyone to use a single version of black means that people will just not bother with your project.
The authors of black just don't understand that it'd be ok to introduce new rules to format new syntax, but it isn't ok to just change how previous things work.
This is mostly nonsense and FUD. We have virtualenvironments, requirements files, setup.py with extra_requires that can all be used to manage versions without relying on the particular packages installed on an OS. Most people contributing to open source would be familiar with at least some of these methods and if they are not it’s a good opportunity for learning.
And if they are not, then maintainers can pull, run black over the diff, and commit.
CI prevents poorly formatted code from entering main.
The actual changes between black versions of late have been minor at best. You’re making a mountain out of a molehill.
Having a tool that dictates formatting is a lot less oppressive to new developers than 100 comments nitpicking style choices.
> Having a tool that dictates formatting is a lot less oppressive to new developers than 100 comments nitpicking style choices.
Yes, it would work very well if said tool didn't change its mind every 6 months, generating huge commits at every bump
> Most people contributing to open source would be familiar with at least some of these methods and if they are not it’s a good opportunity for learning.
You seem unfamiliar with the fact that other people aren't necessarily clones of yourself and might not behave like you.
> CI prevents poorly formatted code from entering main.
If you run black on CI… which of course I don't since every time they make a new version I'd have the CI start failing.
And no pinning is not a "solution"… it's at best a workaround for badly written software.
> The actual changes between black versions of late have been minor at best. You’re making a mountain out of a molehill.
If you have 10 lines of code, I guess your diff can't be more than 10 lines. If you have more than 10 lines…
I’m working in a Python code base with multiple millions of files and not for the first time. It’s not the problem you make it out to be. The changes between black versions have been almost unnoticeable for years.
since developing in python should be done in a virtual env to start with, I fail to see how this will be any problem.
The pre-commit documented version of black will be installed in the venv of the project, problem solved.
pre-commit is very useful, in my opinion. When organising code from a lot of Python developers at least, getting the boring stuff like formatting, import ordering, linting, mypy etc. sorted is a time saver.
Do you know how slow all of that is? Do you want to run all of that per every commit? The result would be people making a monocommit rather than incremental commits that is easy to review one by one.
Two developers on the same python project should also use the same version... with poetry it is straightforward to keep track of dev dependencies. Reorder python imports is an alternative for isort: https://github.com/asottile/reorder_python_imports
> Two developers on the same python project should also use the same version
Why? It is expected for the thing to run on different python versions and different setups… what's the point of forcing developers to a uniformity that will not exist?
It's actually better to NOT have this uniformity, so issues can get fixed before the end users complain about them.
> So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
Any team of developers who aren't using the exact same environment are going to run into conflicts.
At the very least, there must be a CI job that runs quality gates in a single environment in a PR and refuses to merge until the code is correct. The simplest way is to just fail the build if the job results in modified code, which leaves it to the dev to "get things right". Or you could have the job do the rewriting for simplicity. Just assuming the devs did things the right way before shipping their code is literally problems waiting to happen.
To avoid CI being a bottleneck, the devs should be developing using the same environment as the CI qualify gates (or just running them locally before pushing) with the same environment. The two simple ways to do this are a Docker image or a VM. People who hate that ("kids today and their Docker! get off my lawn!!") could theoretically use pyenv or poetry to install exact versions of all the Python stuff, but different system deps would still lead to problems.
Would you please stop breaking the site guidelines? You've been doing it repeatedly, unfortunately. We want thoughtful, curious conversation here—not flamebait, unsubstantive comments, and swipes.
A lot of modern open source projects include a lock file or some other mechanism that ensures that all contributors use the same versions of certain key tools. Obviously there are still going to be some differences in the environment, but for things like formatting, linting, etc, it's generally fairly easy to lock down a specific version.
In Python, the easiest way to achieve this is using Poetry, which creates a lock file so that all developers are using a consistent set of versions. In other languages, this is generally the default configuration of the standard package manager.
The popular Rust tool "ripgrep" uses a lock file for development (you can see it in the GitHub repo), and yet is in the official repositories for homebrew, various Windows package managers, Arch, Gentoo, Fedora, some versions of openSUSE, Guix, recent versions of Debian (and therefore Ubuntu), FreeBSD, OpenBSD, NetBSD, and Haiku.
With all due respect, I don't think you're correct.
And how much rust software is packaged in distributions? Almost none. They haven't figured out the procedures, because distributions really really don't want pinned stuff around.
Homebrew, windows, arch all have very very relaxed processes to enter. There is no QA, you can just do whatever you want. I mean more like Fedora and Debian.
Bottom line is that the lock file in ripgrep's repo hasn't prevented it from being packaged. And I haven't heard of any distro maintainer complain about any lock file in any Rust program ever. So you're just plain empirically wrong about lock files preventing Rust programs from being packaged.
You've now moved on to talking about something else, which is "how much Rust software is packaged." Well, apparently enough that Debian has Rust packaging policy[1]. I'll give you one guess at what isn't mentioned in that policy. Give up? Lock files!
How Linux package managers handle these newer languages with their own package managers (including rust) is an ongoing pain point. Here’s an article from 2017 about it, and I don’t know if things have improved:
I didn't say it wasn't a pain point or that there weren't challenges. I said I don't hear about people complain about how Rust programs are packaged. Not that the packaging of Rust programs (among others) itself doesn't present interesting challenges depending on the policies of a particular distro. Archlinux, for example, has far fewer problems than Debian because of differences in policy.
The poster I was responding to was literally posting false information. I'm correcting it. This doesn't need to turn into a huge long sprawling discussion about packaging Rust programs. The main point that I was making is that lock files do not prevent Rust programs from being packaged. bombolo then went off on their own little tangents spouting nonsense without bothering to acknowledge their mistake.
I contribute to packaging. But thanks for teaching me about something I know already.
Now try to get something using an obsolete version of some python module into Fedora or Debian and let me know how it goes… It would not be accepted as it is. It'd be patched to work with a current one or just rejected.
I never said a single word about Python. Whether you contribute to packaging or not has nothing to do with whether you're posting false information. If anything, it makes what you've said worse. You should know better.
Just stop spreading misinformation. And the courteous thing to do is to acknowledge an error when it's pointed out instead of doubling down and redirecting as if no error was made.
> Using lock files is a good way to make sure your software never ends up in a distribution and in the hands of users.
> And how much rust software is packaged in distributions? Almost none.
> They haven't figured out the procedures
You're clearly talking about Rust in the second two comments. Your original comment was just a general pronouncement about lock files. You could perhaps be given the benefit of the doubt that you were only thinking about Python, but someone else interpreted your comment broadly to apply to any language with lock files. If you really only meant it to be specific Python, one would reasonably expect you to say, "Oh, sorry, I was only talking about Python. Not Rust. Their situation might be different."
But no. You doubled down and started spouting nonsense. And you continue to do so!
> Where dependency pinning is the norm, there is a culture of breaking API compatibility.
Rust does not have this problem. It is the norm to use lock files for Rust programs, but there is no culture of "breaking API compatibility" without appropriate signaling via semver.
It's easy for you to go off and spout bullshit. You've even been corrected by someone else who maintains distro packages. But it's a lot harder to correct it. You wriggle and squirm and rationalize and deflect.
The distros will eventually stop this dangerous practice of mixing and matching versions for all dependencies. It can only work for a small set of system components, which is what every other OS does.
For people that want to build with the exact set of dependency versions tested by upstream. Just because some distros don't use them doesn't mean there isn't any point.
Distros can keep their own lock file that is based on their own release branch's versions. If it doesn't build, the pkg maintainer will either file a bug report or make a patch, or neither.
But something building doesn't mean that it will work.
There can be changes that are different than function signature changes.
Where dependency pinning is the norm, there is a culture of breaking API compatibility. And you might not have a compiler error to inform you that the API has changed. Sometimes all you have is a commit message.
> All those big changes introduce commits that make git bisect generally slower.
Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes?
> Which might be awful if you also have some C code to recompile at every step of bisecting.
That reminds me, I've got to try out ccache (https://ccache.dev/ ) for my project. My full compile is one minute, but the three files that take longest to compile rarely change.
Perhaps the poster meant that the contents of the commits themselves make bisection slower? By touching a lot of files unnecessarily, incremental build systems have to do more work than otherwise.
Could be? I've not heard of that problem, but I don't know everything.
I'm not used to Python code (which is all that black touches) as being notably slow to build, nor am I used to incremental build systems for Python byte compilation.
And I expect in a project with 2 developers which is big enough for things to be slow, then most of the files will be unchanged semantically speaking, only swapping back and forth between two syntactically re-blackened representations, so wouldn't an caching build system be able to cache both forms?
(NB: I said "caching build system" because an incremental build system which expects time linear order, wouldn't be that helpful in bisection, which jumps back-and-forth through the commits.)
Every file that changed because rules changed, which shouldn't be frequent, I don't remember black changed radically since its creation, do you have an example of some widespread syntax change ?
1) you don't have one black commit for every non-black commit, do you? Because the general best practice is to do like kuu suggested and have a specific black version as part of the development environment, with a pre-commit hook to ensure no random formatting gets introduced.
2) assuming 500 commits in your bisection, that's, what, about 9 compilations you'll need to do, so it will take you 9 hours to run. So even with a black commit after every human commit, that yes, 1 hour more, but it's also only 11% longer.
Even with only 10 non-black commits and 10 black commits, your average bisect time will only increase from 3.6 hours to 4.6 hours, or 30% longer.
I'm curious to know what project you are on with 1 hour build times and the regular need for hours-long bisection search, but where there isn't a common Python dev environment with a specific black version. Are you using ccache and/or distributed builds? If not, why isn't there a clear economic justification for improving your build environment? I mean, I assume developers need to build and test before commit, which means each commit is already taking an hour. Isn't that a waste of expensive developer time?
And, I assume it's not the black formatting changes which result in hour-long builds. If they do, could you explain how?
As I said in other comments, if you try to force contributors to reproduce exactly your local setup, you will be left with no contributors. Which is why you set up a CI to run the tests… because people will most likely not.
As for build times, it was an extreme example. But even an extra step taking 5 extra minutes is very annoying to me…
> if you try to force contributors to reproduce exactly your local setup, you will be left with no contributors. ...
That's not been my experience. To the contrary, having a requirements.txt means your contributors are more likely to have a working environment, as when your package depends on package X but the contributor has a 5-year-old buggy version of X, doesn't realize it, and it causes your program to do the wrong thing.
In any case, your argument only makes sense if no one on the project uses black or other code formatter. Even if you alone use it, odds are good that most of your collaborator's commits will need to be reformatted.
> .. an extra step taking 5 extra minutes ...
How do black reformatting changes cause an extra 5 minutes? What Python code base with only a couple of contributors and no need for a requirements.txt takes 5+ minutes to byte-compile and package the Python code, and why?
Adding 5 minutes to you build means your bisections are taking at least an hour, so it seems like focusing on black changes is the wrong place to look.
None of your comments mention running the full test suite, only build.
When I've used bisection, I've always had a targeted test that I was trying to fail, not the entire test suite. This is because the test suite at the time of that commit wasn't good enough to detect this failure. Otherwise it would have failed with that commit.
Instead, a new failure mode is detected, an automated test developed, and that used to probe the history to identify the commit.
Why are your bisections doing the full suite?
> Black reformatting causes more steps in bisecting
Yes, of course it does. But it's log2(n).
The worst-case analysis I did assumed there was a black commit after every human commit. This is a bad practice. You should be using black as a pre-commit hook, in which case only your new collaborator's commits will cause re-formats. And once they are onboard, you can explain how to use requirements.txt in a virtualenv.
If only 10% of the commits are black reformattings, which is still high!, then a bisection of 100 human commits (plus 10 black commits) goes from about 6.64 tests to 6.78 tests, which with a 5 minute test suite takes an additional 42 seconds.
If it's 3% then your bisection time goes up by 13 seconds.
If you are so worried about 13 seconds per bisection then how much time have you spent reducing the 5 minute test suite time? I presume you run your test suite before every commit, yes? Because if not, and you're letting CI run the test suite, then you're likely introducing more commits to fix breaking tests than you would have added via black, and taking the mental task switch hit of losing then regaining focus.
> This is a bad practice. You should be using black as a pre-commit hook
I would reject such commits in review.
A human might add one or two items to a list and black might decide it's now too long, and make 1 line into 10 lines.
Now I have to manually compare the list item by item to figure out what has changed.
So I normally require formatting to be done in a separate commit, because I don't want to review the larger than necessary diffs that come out doing it within the same commit.
> A human might add one or two items to a list and black might decide it's now too long, and make 1 line into 10 lines.
A human might add one or two items to a list, decide it's now too long, and make 1 line into 10 lines.
Including the same hypothetical first contributor you mentioned earlier, who you think will find using requirements.txt as being too big a barrier to entry.
Onboarding occurs either way.
I get that you don't like using black - and that's fine! I don't use black on my project either.
But it seems like you're trying to find some other reason to reject black, and constructing hypotheticals that don't make any sense.
Just say you don't like black's choices, and leave it at that.
> I get that you don't like using black - and that's fine! I don't use black on my project either.
Well according to this comment, it's because we are noobs: "the people that disagree just haven't figured out that they're wrong yet"
> But it seems like you're trying to find some other reason to reject black, and constructing hypotheticals that don't make any sense.
After the n-th time I have to re-setup the entire virtual env on a different branch just to re-run black and isort to backport a fix to a release branch… it does get tiring.
I presume most people here just do websites and don't really have a product that is released to customers who pay to support old versions for years, so black changing syntax is a 1 time event rather than a continuous source of daily pain.
But it seems the commentators here don't have the experience to know there might be a use case they didn't think of.
> isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before.
This is not isort! isort has never done that. And it has a formatting guarantee across the major versions that it actively tests against projects online that use it on every single commit to the repository: https://pycqa.github.io/isort/docs/major_releases/release_po...
Hi! I said this with more certainty than I should have. Software can always have bugs! For reference, I wrote isort, and my response came from the perspective that I have certainly worked very hard to ensure it doesn't have any behavior that is random or non-deterministic. From your description, it sounds like someone may have turned on force-alphabetical-sort (if this is in a single project). See: https://pycqa.github.io/isort/docs/configuration/options.htm.... You can do `isort . --show-config
`, to introspect the config options isort finds and where it finds them from within a project directory. The other thing I could think of, is coming from isort 4 -> 5, I wouldn't think it would fully ignore import groupings, but maybe it doesn't find something it used to find automagically from the environment for determining a first_party import. If that's the case this guide may be helpful: https://pycqa.github.io/isort/docs/upgrade_guides/5.0.0.html. If none of this helps, I'd be happy to help you diagnose what your seeing.
> So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
You should never develop using the system Python interpreter. I recommend pyenv [0] to manage the installed interpreters, with a virtual environment for the actual dependencies.
> Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
What's the alternative? YAPF is even worse - it will flip flop between styles even on the same version! Its output is much less attractive, and there are even some files we had to whitelist because it never finishes formatting them (Black worked fine on the same files).
Not using a formatter at all is clearly worse than either option.
Because some people are really bad at formatting code manually and constantly nitpicking them about it is both tedious and antagonistic. Its much better for a faceless tool to just remove formatting from the equation entirely.
I think the sane part of the software engineering world has realised that auto-formatting is just the right way to do it, and the people that disagree just haven't figured out that they're wrong yet.
Maybe you meant "why is Black specifically better than no autoformatting, given that it isn't perfectly stable across versions?" in which case the answer is:
a) In practice it is very stable. Minor changes are easily worth the benefits.
> the people that disagree just haven't figured out that they're wrong yet.
This is unnecessarily confrontational. Please read my other comments where I consider the extra effort that automatic formatting causes for code reviews.
> In practice it is very stable.
It has never happened to me to upgrade black and have it not change opinion about black formatted code.
> Minor changes are easily worth the benefits.
It doesn't matter how minor they are. A 1 bit difference is still going to fail my CI.
> You can pin the version!!
I usually do, but working with old releases that must be maintained, mean that I can't cherry pick bug fixes from one branch to the other, because black fails my CI.
As I said… because a 1 word change easily ends up changing multiple lines and from the diff it's not clear it's just a 1 word change. So… extra effort.
> Right but you have a pre-push hook to format the code using the same version of Black as is used in CI. Then CI won't ever fail.
No I don't. My release branch and old-release branch use different versions. Such a thing would need to understand which base branch I'm working on, or recreate the venv continuously.
> Cherry pick, then run Black. Sounds like you have a very awkward workflow to be honest.
Seems to me you don't support older versions, in which case everything is easy.
I love this template as well, and wholeheartedly recommend it. There are a couple things you probably don't need (click and nox, for instance, seem only useful if you're really building a couple specific things) but the gestalt of it is really strong. The [article series](https://medium.com/@cjolowicz/hypermodern-python-d44485d9d76...) that spawned the template is worth reading in full.
I would go so far as to say that the hypermodern template, nomenclature aside, is strictly better than the recommendations that the OP put forward both here and in the previous essay on dependency management. Poetry and ruff, for instance, are both very good tools — and I can understand _not_ recommending them for one reason or another but to not even mention them strikes me as worrisome.
I don't work on large python projects, mostly just small scripts that need to work well (integrating with a 3rd party rest api is a good example). I don't do CI or unittests but I use git. This is because it takes time and honestly no one outside of myself would care for small stuff like that. But I do run autopep8 and pylint it (I ignore stuff like line being too long,broad exception handling or lack of docs).
My concern is a) It needs to be reliable (don't wanna spend a ton of time chasing bugs later on) b) How can I write the actual code better? I see what pro devs write and they use smarter language features or better organization of the code itself that makes it faster and reliable, I wish I could learn that explicitly somewhere.
I mean, just the 2.7->3.0 jump was big for me because since I don't code regularly that meant googling errors a lot basically. Even now, I dread new python versions because some dependency would start using those features and that means I have to use venv to get that small script to work and then figure out how to troubleshoot bugs in that other lib's code with the new feature so I can do a PR for them.
I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly. Those are just not suitable for people whose day job is not coding and migrating to new versions, supporting code bases, messing with build systems, unit tests, qa,ci,etc... coding is a tool for me, not the centerpiece of all I do. But python is still great despite all that.
> I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly.
What do you mean by "drastic" features "quickly"? Python releases new version once a year these days, and upgrading our Django-based source code with 150 dependencies from 3.4 to 3.11 literally meant switching out the python version in our CI configuration and README.rst every once in a while, no code changes were necessary for any of those jumps...
Our developer README also contains a guide how to set-up and use pyenv and it's virtualenv plugin which makes installing new python versions and managing virtualenvs easy, just pyenv install, pyenv virtualenv, pyenv local, and your shell automatically uses the correct virtualenv whenever you're anywhere inside your project folder...
jumping to python3 was big, but you had plenty of time to prepare for that and plenty of good utilities to make the jump easier (2to3, six, ...). python2.7 itself was released 18 months after python3.0, and by the time python2.7's support ended, python3.8 was already out...
Firsr, to this date, stuff I absolutley need that is in 2.7 i have to either try to fix or venv or somehow get it to work is one of my biggest headaches (Not my code).
Second, yes, all you have to do is switch out the python version to upgrade but let's say you start using f-strings that means all of your users (doesn't apply to django since it is server software) have to upgrade to the right python version including all the deps. But what if your project is a library? That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
New versions should come out no more often than every 3-4 years imho and even then every effort should be made to have those features backward compatible like have a tool that will degrade scripts to be usable on a previous language version.
> Firsr, to this date, stuff I absolutley need that is in 2.7 i have to either try to fix or venv or somehow get it to work is one of my biggest headaches (Not my code).
2.7 was supported for 10 years and it's support ended 2 years ago. There's been ample time to upgrade the code or look for an alternative. If I "absolutely needed" to use a piece of code that I didn't write, is for an unsupported platform and is itself unsupported, I'd absolutely find the time for it. As a developer if I use a library that hasn't been touched for 3 years it's a red flag and I start to look for alternative libraries or forking the code.
> That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
if your distro doesn't support the latest python version you're probably on a very old distro. For example python3.11 installs fine on all supported versions of Ubuntu (18.04+) and Debian (10+) and both Windows (8.1+) and macOS (10.15+). And python3.9 installs fine even on centos7 (released in 2014) and still supports the vast majority of python libraries.
If you're on an OS nearing or past its end of support, you can't reasonably expect all the latest software to work on it. And it's usually fine to just use an older version of python / libraries until you're ready to update.
> New versions should come out no more often than every 3-4 years imho
If new versions came out every 3-4 years, that would mean they would have more drastic changes, because the smaller changes would accumulate over that duration. The longer the "new features" are out, the longer users have to upgrade their system and the longer developers can take getting used to them.
But in the end, it doesn't really matter how often a new version comes out but rather how long the old versions should be supported, right? And I think it's up to the library authors to decide how long to support older versions, not the authors of the programming language.
> ...I'd absolutely find the time for it. As a developer if I use a library that...
Like I said, my day job isn't being a dev which means time for that is rare.
Languages are not user software, they should never be deprecated. I get deprecating standard libraries bur under no context is it ok to deprecate a whole language. Yoh can freeze development and only perform security updates to the interpreter but there is no need to deprecate a language. It is a betrayal of the trusr users put into python when they invested time on it and this is exactly what I mean by avoiding rapidly changing languages. They don't care one bit that people are using their language, they treat like any other software that gets supported and discarded. C89 is still supported! People write new stuff with it. You know why? Because there is nothing to support, just parsing and compiling of a langauge. No new features need to be developed and bugs should be accepted instead of fixed. The interpreter should be available for download and use on any platform for as long as even one guy is using the language.
> you're probably on a very old distro.
Maybe, I usually go for debian but I have run into this issue and it becomes a dependency nightmare on anything that needs the old python version (in the package manager dependency resolver not in python).
> If new versions came out every 3-4 years, that would mean they would have more drastic changes
That's fine, because there would be less versions. All changes are drastic changes from the perspective of someone that is having to google random python errors to figure out what broke and how to fix it. At least it won't be a constant nightmare fixing problems made by the langauge itself in addition to the 3rd party code and your own code. The frequency of how many bugs you have causes by those 3 cause categories should in that increasing frequency. I should not have lnaguage version bugs more often than bugs in my own code.
> But in the end, it doesn't really matter how often a new version comes out but rather how long the old versions should be supported, right? And I think it's up to the library authors to decide how long to support older versions
It does matter because most devs that code for a living like to tinker with new flashy versions so each version of a library they release is that much more prone to requiring newer python versions. The less frequent python releases, the more they will use the current python features before introducing version breaks.
I really think it is a developer culture problem at the end of the day where because you are writing foss code, you don't care about the experience of those who depend on your code.
> Even now, I dread new python versions because some dependency would start using those features
If a dependency breaks compatibility with earlier Python versions because the author wants to use a fancy new feature is not really the fault of Python, is it? Library authors should target the earliest supported Python version they can.
Being backwards compatible (at which Python has been doing a good job since the 2->3 fiasco) is one thing, but trying to be forwards compatible is something else.
Are you suggesting that Python developers should only ship bug fixes so that Python 3.0 can still run code written for Python 3.11?
This isn't a general problem, but I have been seriously burnt by a change from a minor version upgrade.
In 3.8 someone decided that they didn't like the way people were excepting the Exception for cancelled asycnio tasks. So they changed the cancelled task exception to inherit from base exception instead of exception. This meant a bunch of well used libraries immediately had a load of subtle bugs that in normal operation just didn't happen. I can't remember the exact details but I think when the bug did happen the task queue would just continue to grow until we ran out of memory.
This change wasn't a bug fix, more an optimization or an attempt to get people to code a certain way.
I'm all in favour of bug fixes, but Devs shouldn't have to worry about minor upgrades breaking everything.
It doesn’t mean it’s Python’s fault, but it fosters a culture where Python developers who regularly follow the language and are some of the ecosystem’s biggest authors are enticed to trying out the fancy new features (even if the old way still works) because “this is cleaner, this is how I want to do things from now on”.
You should not pin the public requirements that get uploaded with a library (listed in setup.py, setup.cfg, or pyproject.toml), since that will restrict your downstream users, leading to version conflicts and persistent security vulnerabilities.
But it’s totally reasonable to pin the private requirements that you develop it against (listed in requirements.txt, poetry.lock, or similar), updating them every so often during the course of development, so that contributors can use a consistent set of tools.
> For your dependency/versioning issue, use a virtualenv per-project and pin your dependency versions in requirements.txt
requirements.txt is not uploaded to PyPI and has no effect on your package’s dependencies when a user installs it (leaf package or no). It’s only used for developing the package itself, typically in a unique virtual environment.
Agree in principle, but I'm giving advice to someone who programs on occasion and is primarily concerned with their programs breaking due to dependency version upgrades when they come back to them after a little while.
My advice would be to use stuff that is in distributions… it hopefully (not necessarily) is maintained by less noob people who don't break API all the time.
Not agreeing/disagreeing with the message, but the style of writing here is quite nice. It's focused, reasoned, and doesn't make too many assumptions about your tools and environment--and I appreciate that acknowledgment.
One thing that is underestimated is keep the tools version in sync between your app dev dependencies and pre-commit. This also includes plugins for specific tools (for instance flake8). A solution would be to define the hooks in pre-commit to run the tools inside your venv.
About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.
I disagree with this assessment on running a static type checker, although I will admit, every update of python over the past 3 years seems to add more and more typing changes which tends to force global typing updates (looking at you Numpy for python 3.12!)
When python converges on consistent typing across its extended numpy and pandas ecosystem, I believe we will be able to move towards a fully JIT'd language.
> I believe we will be able to move towards a fully JIT'd language.
Unless they actually go ahead with the deferred evaluation of types (PEP 563), make all types strings at runtime and make it impossible to know which type they actually are. :)
What's the current state of the art of managing multiple virtual environments, running tests and running your application?
On Ubuntu and Windows I use Poetry [0], and it works, although it has (had?) some quirks during the installation on Windows.
I liked its portability and lockfile format though.
A few years ago I used conda [1], which was nice because it came batteries included especially for Deep Learning stuff.
I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi.
And then there are also Docker Images, which I use if I want to give an application to somebody that "just works".
I use pip-tools to build a requirements.txt file from a requirements.in file. It does basically the same as poetry, but more manually. For me that's good because one of the application has a lot of requirements, and it needs to be deployed on systems with different Python versions, and the requirements need to be packaged along with the application because the servers have very limited internet access. So as long as Poetry doesn't add good support for multiple python versions and/or easy packaging of all dependencies, it isn't worth it for me to do the migration.
I'm liking PDM for a while now. Quicker than Poetry and built according to the Python package spec in mind and not as an afterthought. While it was originally meant to work with PEP 582, it works with virtual environments too (now default).
>I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi.
Agreed. I like docker images for smallish portable scripts. At home I can develop on my Mac and port it to a Raspberry PI or another x86 Windows/Linux box.
Planning on running a docker swarm with a few Pi’s to see how it works.
I wish VSCode would figure out that ExampleModel.objects.first() returns ExampleModel or None or ExampleModel.objects.filter() returns an iterable of ExampleModel. Has anybody gotten this working, automatically or manually annotating?
You can annotate the manager and get some typing help in the editor. And there’s django-stubs which helps a little when running mypy. It’s not as good as pycharm though.
https://github.com/charliermarsh/ruff
It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.