1. 30

Hello,

Would love to hear from community on your favorite python codebases and how can you become a better python developer.

I am currently a bit rusty when it comes to writing python but I would love to further improve my skillset

  1.  

    1. 13

      I would recommend 4 repositories:

      • Pydantic: It will show you all the “magic” of Python. Their codebase is typed, so it’s a mix of good practices. They have tests, a good CI setup, and docs. You can get some inspiration on how a “production-ready” Python package could look. Also, the amazing part is that they have Rust integration, so you can see how this happens in real-world scenarios - it’s a win-win.

      • FastAPI: A very popular web framework that also uses Pydantic. They have a very good codebase with tests, so you can take a look at how you need/can structure them for web-like projects. They have translations for their docs, so it also can be interesting to see how it’s done on a production scale.

      • SQLAlchemy: A very good and authoritative Python ORM. You can see how the ORM works under the hood, so while reading this codebase you can learn a lot about SQL. If you switch to some older version prior to typing, you can see how it was done in the good old days.

      • Kedro: A top-notch quality codebase where you can learn about how data engineering frameworks are built. They have very vast CI pipelines and heavy use of Click plugins and package optional dependencies, so you can see how a package can be scaled.

      1. 2

        3 repositories

        :-)

        1. 4

          thanks ! I wanted to write about 3 but couldn’t stop :)

        2. 4

          You know what they say about the 2 hardest problems in computer science XD

    2. 11

      Someone asked me this question just yesterday in a Python course I was teaching.

      I told them:

      1. look at the design (but not code) of long-time, major, popular, high-velocity third-party libraries that emerged after Python 3 was released
      2. look at anything written by Dave Beazley
      3. look at anything (greenfield) by Tom Caswell

      Regarding ①, if you look at the code for NumPy, you’re going to get lost in lots of mechanical details. It’s an extremely successful tool, but it’s probably not how we would write Python code today. If you look at the design and user-facing API of NumPy, you’re going to see how things were designed in the early Python 2 era, which is missing many key features available today. This is the same for Matplotlib (which has set_ functions, because @property didn’t exist when Matplotlib was first created.) However if you look at pandas, you will see a tool that, as a consequence of its extreme popularity, was forced to evolve a very coherent and contemporary design, marred only by extreme inconsistencies at its most superficial level (e.g., nothing is spelt consistently—.swaplevel vs .reorder_levels) Skip PyMC as a reference—the API is just bizarre. (Use PyMC if you want to do Bayesian statistical modeling.)

      Regarding ②, read Dave’s recent book: Python Distilled Anything Dave still maintains or has recently released will be good. (SWIG was a long time ago.) He’s one of the few programmers I have met who “just gets it”—his code is lucid, clear, and direct.

      Regarding ③, Bluesky Project is a constellation of projects for data acquisition and management that is in use at a couple of x-ray beamlines in the US. I am not a physicist, and I don’t work on at a beamline, but I do a lot of Python and other training for scientific research facilities. I have taught two classes on the bluesky and databroker libraries from Bluesky Project to physicists. These classes weren’t about the physics—I have no background in that; they were about the software. In answering attendee questions, I have written lines of code that I have never written before in my life, using a library that is not in my field of expertise or something I use, and yet I could predict what the line of code would do before executing it… and I was right every time. (e.g., “How do I see if my experiment was in the output payload?” “Well, the example shows they implemented __iter__ and __getitem__ on the object representing your results, so it stands to reason they implemented __contains__ to determine membership using the __getitem__ key. … And that works.”) In my view, the marker for well-designed (Python) code is whether someone broadly familiar with the ecosystem can correctly intuit the operation of new libraries, using applicable general knowledge.

      Unlike what other contributors have shared, I would skip Pydantic and FastAPI. I like FastAPI, and just used it very recently in a project, but the design itself is focused very heavily on convenience in a way that other projects may not be. As a consequence, the underlying code will require contortions (e.g., the use of advanced features) to support these conveniences. If I could, I would skip Pydantic entirely—I wouldn’t even use it, if I could avoid it. It’s the wrong approach solving an uninteresting problem.

      The FastAI library moves with high velocity, so it may have improved since I looked at it some years back, but, if not, I would stay far, far away; it’s full of bizarre, wholly unnecessary choices in the service of a design overly-focused on an idiosyncratic notion of convenience.

      People seem to like Textual, and they put out some great demos. I haven’t looked at the code, but I have looked at their user-code samples, and it looks like they picked the wrong metaphors completely. It might end up being a good example of how a project can be popular (and enable useful work!) irrespective of its technical merits.

      I have a YouTube channel with instructional material where I cover some of these things. I have also given some talks at PyData and other conferences about these things.

      I think contemporary, “fluent” Python code is typified by the following things (among others):

      • precise, correct usage of the Python “vocabulary” offered by the data model
      • use of ‘iteration helpers’ (and, e.g., the itertools module); “every line of Python code should be a sentence (which explains what you are trying to accomplish without burying the reader in mechanical details)”
      • design around ‘restricted computation domains’
      • ‘value disintermediation’ (which is a particular distinguishing mark of Python 2 vs Python 3 code); ; we can associate this with use of things like enum.Enum

      The use of PEP-484 type hinting is on this list. Use them if you want; sometimes they are helpful, sometimes they’re just noise. Similarly, gratuitous use of new features is obviously never the mark of good code.

      Here is a brief example of the use of ‘iteration helpers.’ (The other items above would require longer explanation.) Consider that the improvement in the below is not merely a superficial issue of performance or number-of-lines; the latter formulation is actually more nearly correct!

      from random import Random
      from itertools import accumulate, pairwise
      from operator import add
      
      rnd = Random(0)
      
      # left scan; cumsum; random walk
      values = [*accumulate((rnd.randint(-1, +1) for _ in range(5)), add, initial=100)]
      
      # without iteration helper—noisy, mechanical, subtly wrong
      prev_value = None
      for curr_value in values:
        if prev_value is None:
          prev_value = curr_value
          continue
        print(f'\N{mathematical bold capital delta}: {curr_value - prev_value = :>2}')
        prev_value = curr_value
      
      # with iteration helper
      for prev_value, curr_value in pairwise(values):
        print(f'\N{mathematical bold capital delta}: {curr_value - prev_value = :>2}')
      

      In the end, though, never forget the difference between good code and bad code is always been the same no matter the language:

      Good code is code I wrote; bad code is code you wrote.

      1. 2

        If I could, I would skip Pydantic entirely—I wouldn’t even use it, if I could avoid it. It’s the wrong approach solving an uninteresting problem.

        What do you think the correct approach to this problem is?

        ‘value disintermediation’ (which is a particular distinguishing mark of Python 2 vs Python 3 code)

        I haven’t heard this term before. Could you elaborate on it?

        He’s one of the few programmers I have met who “just gets it”—his code is lucid, clear, and direct.

        His talks are a joy to watch and his powerpoint on coroutines was really eye opening for me.

        1. 2

          ‘value disintermediation’ (which is a particular distinguishing mark of Python 2 vs Python 3 code) I haven’t heard this term before. Could you elaborate on it?

          I may have made this term up (like “restricted computation domain,” “nominal decomposition of a bounded modality,” &c.) However, it describes something I have definitely seen.

          Basically, under the assumption of the ‘restricted computation domain’ approach to Python performance (e.g., NumPy, pandas, Polars, &c.,) our Python programmes are bifurcated (at the extremes) into the “programme structuring” domain and the “computational domain.” (This is why, for example, I think things like __slots__ are just not very interesting at all.)

          This then presumes that the directly-manipulable entities in your Python code are high-level “programme structuring” entities. They are relatively large, relatively long-lived, and relatively few in number. As a consequence, they are well suited to management by the garbage collector (as part a large object graph… with circular references!)

          This, in combination with the use of things like enum.Enum leads to the so-called “value disintermediated” approach. In essence, we want the operation of our Python programme to be on high-level objects rather than on values.

          So, for example, if we have some generator coroutines representing state machines for some workflow process, we might have code that looks like:

          from enum import Enum
          from dataclasses import dataclass
          from asyncio import run
          
          Role = enum('Role', 'Admin Supervisor Standard')
          
          @dataclass
          class User:
            role : Role
            ...
          
          def workflow_abc(user : User):
            yield
          def workflow_def(user : User):
            yield
          
          active = defaultdict(set, {})
          user = User.from_name(name)
          active[user, workflow_abc].add(workflow_abc(user=User))
          active[user, workflow_def].add(workflow_def(user=User))
          

          Note the absence of str in lieu of enum.Enum, that we pass rich objects around, that we have dict where the keys are functions and the values are set of generator coroutine instances. We are strenuously avoiding having mere int or str values stand in the way of the richer (often symbolic) entities we want to manipulate—“value disintermediation.”

        2. 1

          If I could, I would skip Pydantic entirely—I wouldn’t even use it, if I could avoid it. It’s the wrong approach solving an uninteresting problem. What do you think the correct approach to this problem is?

          I’ve always been skeptical of “single-field, discrete validation,” since it routinely feels like the kind of problem that is tedious to do, interesting to try to automate, but largely not very useful. Most moderately complex business use-cases I have seen involve multi-field, continuous validation—there are gradations of validity, which are determined though some holistic inspection of multiple fields.

          It doesn’t help that the behaviour of the pydantic.BaseModel diverges so dramatically from the expectations of a normal Python class (or even a dataclasses.dataclass,) and it seems presumptuous that my data validation library—which is the least important and least interesting part of my code!—should be dictating design decisions.

          Consider:

          from pydantic import BaseModel
          
          class AMeta(type): pass
          class A(BaseModel, metaclass=AMeta): pass # fails
          

          Sure, we could have AMeta derive from the BaseModel metaclass, but then I would be constrained by having to compose with some internal library code that could change out from under me within a minor-release.

          Consider:

          # company/facility-level library code
          class Base: pass
          class Derived(Base): pass
          
          # group/user-level library code
          class Implementation(Derived): pass
          

          I am often already constrained in what I am allowed to do, through the natural organisational hierarchy of my code. If a tool is used at levels above me, or if I want to use a tool only within the level I control, then I want that tool to be noninvasive and nondisruptive. This is why, for example, I would want to prefer a class decorator over a metaclass—it’s a much “lighter touch.”

          If it’s a validation or de/serialisation tool, I would want to surgically implement the behaviours I need for these tasks in a way that does not create new constraints. I want constraints to come from the interesting parts of my code, not, like, validation and de/serialisation!

          I don’t want the introduction of such a tool to change fundamental behaviors in any way that I might notice and have to care about!

          e.g.,

          from pydantic import BaseModel
          
          class T(BaseModel):
              _x = [1, 2, 3]
          
          obj0, obj1 = T(), T()
          assert obj0._x is not obj1._x
          

          Unfortunately, the lighter and less intrusive we make the implementation… well, the less magical, convenient, and useful it becomes under the above design approach. Therefore, I suspect the problem is the design approach itself.

          1. 1

            Most moderately complex business use-cases I have seen involve multi-field, continuous validation—there are gradations of validity, which are determined though some holistic inspection of multiple fields.

            TBH, this sounds like where algebraic data types are useful in certain statically types languages.

            If a tool is used at levels above me, or if I want to use a tool only within the level I control, then I want that tool to be noninvasive and nondisruptive. This is why, for example, I would want to prefer a class decorator over a metaclass—it’s a much “lighter touch.”

            Unfortunately, the lighter and less intrusive we make the implementation… well, the less magical, convenient, and useful it becomes under the above design approach.

            This is a very reasonable take. It seems like the more powerful the abstraction the leakier it is.

      2. 2

        Regarding ②, read Dave’s recent book: Python Distilled Anything Dave still maintains or has recently released will be good. (SWIG was a long time ago.) He’s one of the few programmers I have met who “just gets it”—his code is lucid, clear, and direct.

        Are you saying SWIG is bad, just hard to read, unrepresentative of python good habits, or …? While I wouldn’t hold it up as a paragon of easy reading, I’d still call it really good, think it’s aged very well, and if you need to read python FFI stuff, it’s honestly one of the first places I’d send you. I’m sure I’m biased. My first python swig code is about 20 years old, and I’ve probably maintained more python swig code than your average python liker. I’m only curious because you drew a line between SWIG and Dave’s other stuff.

        1. 1

          SWIG has a reputation, deserved or not, and for the purposes of the question we’re answer, I don’t think it’s worth distracting from Dave’s other, more recent works. I don’t think he’s contributed to SWIG in over ten years.

          Personally, I think SWIG is a product of its time, and if you talk to Dave, he’ll admit that he never envisioned it taking off like it did and never imagined it would be used in some of the ways it is.

      3. 2

        This is a great advice and also your Youtube channel has some great content. I did subscribed to your channel.

        Lastly I wanted to ask how long have you been teaching Python

    3. 6

      They’re mostly on the smaller side, but my go-tos these days are usually the encode codebases (e.g. httpx), and @hynek’s codebases (attrs / structlog might be the most widely used, but stamina and svcs are equally neat).

      1. 2

        Big +1 for @hynek’s work. Beyond the code itself, between their design and documentation, these libraries give a user a good feeling for why they are doing things a certain way.

    4. 4

      I too am very interested in this! I am writing a lot of Python these days and I can escape the feeling that it’s non-idiomatic and gross, even as it passes the various linters and such I have configured.

    5. 3

      I have learned a lot by browsing the GH repos of packages I frequently import. I’ve recently delved quite a bit into the hrequests and undetected-chromedriver repos partly to deepen my understanding of what they’re doing but certainly also to sample other people’s programming styles and techniques. If you find yourself importing certain 3rd-party packages a lot, browse their GH repos!

    6. 3

      Django is a great study case for a large project that has had a large number of contributors

    7. 2

      I’ve read this the Paasta codebase a little bit and thought it was fun. Decently documented, README has some videos and talks about it as well.

      1. 2

        Nice suggestion! But I got nerd-sniped at the Paasta README, which shows how much glue and duct tape they brought together for this project :D

    8. 2

      I remember Mercurial was often given as an example of good Python source code. For a while I practiced touch typing by typing out it’s source code. Just checked and you can still do that: https://typing.io/lesson/python/mercurial/merge.py/1

      1. 1

        However, I believe hg was fairly slow/late to move off python 2… so be aware of that. Unless you’re looking at a recent version, it’s likely to have a lot of 2/3 compatibility overhead and/or maybe not be representative of what you’d write today because of changes and new features in the language.

    9. 1

      I haven’t touched Python in a long time, but I remember the requests package being particularly well written. :)

      1. 3

        Actually, it’s not so great. See the last month discussion about it: https://lobste.rs/s/czcfzg/retrospective_on_requests

      2. 3

        Read an interesting opinion counter-that the other day: https://blog.ian.stapletoncordas.co/2024/02/a-retrospective-on-requests

        Edit: Sniped by @kngl!

    10. 1

      Simon Willison recently suggested this one which maybe database centric, but still: https://simonwillison.net/2024/Mar/19/diskcache/

    11. 1

      The discord.py package is well written and performance-optimized in some places (like slotted classes instead of @dataclass for classes, suprisingly that improves runtime creation a ton). It is also fully typed which is nice.

      1. 2

        (like slotted classes instead of @dataclass for classes

        @dataclass supports slots=True, FYI.

        1. 2

          Only 3.10 and up.

          But in older versions you can use __slots__ with data classes mostly fine, the biggest issue is you lose defaults and fields.

        2. 1

          Interesting, I should test the performance of this versus classic classes with slots. Might post updates soon.