Amber's suggestion seems to be in the same direction (though perhaps not as extreme.)

[0] https://www.slideshare.net/mobile/hsbt/gemification-for-ruby...

The problem is that I cannot count on being able to install new software in many environments.

If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever.

Consequently, the standard libraries need to be very complete and very useful.

And, while people seem to love the Rust approach to libraries, I'm not necessarily a fan. Far too many times I have pulled a library that is "obviously" something that a language should consider to be "standard library" and gotten bitten because it was broken. Only VERY core libraries in Rust are guaranteed to work across multiple architectures and OS's.

I think Rust is probably doing the right thing for Rust as "batteries included" is NOT one of its tenets. However, that doesn't make it right for everybody else.

Can you explain this more? What kind of place do you work? I've had some experience with large, bureaucratic companies, but nothing ever so far as "you can't install any other libraries."

After about 9 weeks of emails, meetings, and pitches, I finally got Anaconda up and running. A week later, I tried to upgrade the 3rd party packages.. No dice. Blocked by the corporate VPN. I'd need the sign off every time I wanted to `pip upgrade` anything

Needless to say, I do not bother anymore.

We have our own development team, our own servers, our own freedom to deliver to clients fast without the hassle of the main corporation. How? We talked to the right persons.

It might me more helpful to think of these types of external factors as fixed points that cannot be moved and just engineer around them.

You'll burn out if you try to boil the ocean on every business process that doesn't seem "logical" from your cursory examination.

And setup.py is a trainwreck, e.g. some packages compile download and compile huge dependencies (e.g. a full Apache httpd...), the default compiler flags may lack all the mandatory security flags (e.g. for using ASLR on python 2.x), or ship their own copy of openssl statically and break your FIPS-140 certification that way...

The corporate world is full of stupid things that will never not change, or take years to change.

In a large company this gives the compliance folks a central place to blacklist packages - along with a trail of what systems have downloaded the package to target for upgrades.

Not OP but same. I'm currently debating with myself whether I should attempt to install PUTTY. Given that port 22 is blocked and it's not needed for my core role it'll be dicey if I get challenged.

Pulling executable code off some repo...no way that is ever officially passing muster. People might do it anyway, but on a personal risk basis.

>Can you explain this more?

Place that are heavy on confidential financial info basically. Practically everything I touch is confidential client data. So employer is naturally jumpy about what's on my laptop software wise.

Ironically the above comes full circle...need putty to get onto a VM in cloud where there are no restrictions and crucially no client data. Nobody cares what I do there - hell they'll even pay for it thanks for MSDN enterprise

Found a work-around though - Google cloud shell being *nix works fine for SSHing about the place. Gets me around the port fw too

Theoretically there's a process for requesting new software and getting it approved, but actually pushing it through requires getting one of my program architects to care enough to file the request (As a mere level 2 engineer all I can do is write it, can't submit), then potentially weeks of followup, for 1 specific version of 1 specific package.

In the case of python packages, perl and the perl packages we need are already approved because a few senior devs got together and pushed them through 10 years ago (was before my time, but I understand it was with quite a bit of arm twisting). It's more time-efficient to just code perl than to fight for python.

It's one of the many reasons I intend to get myself another job for Christmas. :)

As for why the system exists: Cost cutting, in the sense of "the less we invest in infrastructure the more we can divert to sexy hardware for the cameras and shareholder dividends. So long as it's theoretically possible for you to do your work, we don't care how many hoops you have to jump through to do it. And our competition is even worse than us, so we don't have to worry about anyone undercutting."

As a result all our infrastructure is centralized. Programs have to jockey with each other for everything from virtual servers to physical workstations and monitors. Hell the only reason my program has our primary test server is because one of our architects literally overheard a hallway conversation about a program that was spinning down and getting rid of some servers, so he jumped on it.

I worked for a much smaller government contractor, but before I left they were moving to a system where you needed approval from the customer in order to get new packages. (For those who don't work in this field, that means you are actually making a request to the contracting representative from the particular government agency for each package you want.) So it wasn't just in-house bureaucracy in the way of progress, and I generally just went without or wrote my own instead of trying to deal with it.

I'm afraid the standard library has to be aligned with the needs of more normal users who, as already discussed, want to allow libraries to have their own release cycles and to be more "opinionated" and specialized than the standard library would permit.

I'm afraid users dealing with that sort of bureaucracy are much more normal than you think, if not the norm. They're just usually not the types of folks that are hanging around HN, or they're at least less vocal.

As in, it is very difficult to get software installed in general images or on multiuser servers.

There are obviously good reasons for it to be conservative about this.

[0] https://training.kalzumeus.com/newsletters/archive/enterpris...

At which point you have to scale the IT wall all over again if you work at a Fortune 500 company.

I work at a Fortune 500 company, and the only wall I have to scale when I want to use a library that nobody at our company has ever used before, is to get someone to check and approve the license (typically takes 1-2 days), and import it into our code repos.

I mean I see your point, but not everywhere is as bad as you make it seem.

You could install packages with `pip install --user` to have them install under your home directory.

> You could install packages with `pip install --user` to have them install under your home directory.

You might find that the issue isn't necessarily always technical (ie permissions), but policy. A lot of places don't let you arbitrarily download software off the internet onto a system. Pp's point is that having a full feature stdlib let's you only need to crank the policy/approval process once, rather than once for every dependent lib.

The approach Ruby is taking with gemification and default and bundled gems for standard libraries is equivalent to the traditional standard library if you can't install updates, but superior in other cases.

In this case, even If you can't upgrade, you still get the same libraries but perhaps older versions.

In Rust, libstd is mainly for interfacing with the compiler and providing interoperability between packages (crates). The wider crate ecosystem is the real standard library, since external crates are as easy to use as the standard library.

For example, the libstd doesn't even have support for random number generation. There's a rand crate, which is now on 6th major breaking version. That's perfectly fine, because multiple versions can coexist in one program, and every user can upgrade (or not) at their own pace. And the crate was able to refine its interface six times, instead of being stuck with the first try forever.

As to why this is the case, I think maybe this is enabled by Go's backward's compatibility focus and encouragement to upgrade early and often, and the community's focus to utilize small interfaces sometimes from the stdlib itself, like io.Reader and io.Writer, or http.Handler. Added to that w.r.t. using the latest and greatest, most Go users frequently are using the latest version of Go even in production (per the go experience surveys).

I am sure it also helps that Google pays people to develop, maintain, and improve the stdlib.

    CompressedSize     uint32 // Deprecated: Use CompressedSize64 instead.
    CompressedSize64   uint64 // Go 1.1

    // Deprecated: HeaderMap exists for historical compatibility
    // and should not be used.

Sometimes improvements are not worth the cost of deprecation and replacement, so things are just left as they are. For example, an HTTP interface designed for request-response HTTP/1 works for stream-oriented HTTP/2, but support for push, prioritization and custom frames is bolted on. Packet-oriented HTTP/3 will add even more stuff that will have to be retrofitted somehow to the old model. Libraries can come and go, but std can't just throw away an old interface and start over.

- What would you define as a while? I think Go has done well with it, given that it is almost 10 years old now at this point, and in reality, was in development internally at Google years before that.

- Do you feel like an interface being frozen forever is particularly a bad thing in of itself? What if the interface does a really good job describing the thing, whatever it may be? For example, Go's io.Reader/io.Writer interfaces.

I agree sometimes an interface being frozen is bad, but for example when Go standardized the context package, it simply added "Context" to the existing functions that now take a context (e.g. in the database/sql package, you have the old, Exec, Query, QueryRow functions, and after 1.8 you have ExecContext, QueryContext, QueryRowContext, which some people may view as a reason to have method overloading, but I view as adding better clarity.

Rust libraries should be expected to take a few tries to get right, especially earlier in its lifecycle. There's more possibilities and less experience in the language.

You can see a similar effect in Haskell, which has iterated many basic bits of functionality many times over.

I read some discussion the other day about some ways in which the Any type in Rust isn’t as flexible as it could be, since it is frozen the only way to improve it would be to introduce a new name, such as Unknown. Similarly in C# along with adding async support, the standard library added async versions of many methods, eg Read now also has a ReadAsync partner.

It does seem that having multiple names for basically the same thing adds a small but tolerable level of overhead to a language. At least if as much as possible is moved out, then projects can choose to only use the latest versions, and live in a world as if past versions never existed.

The example of random number generators is a good one, too. There are a lot of applications that require (reproducable!) PRNG sequences and sometimes you have to share PRNGs between modules. Now, looking at the rand crate, I see that the prng part of it was recently mucked around with. If I have two 3rd party modules that I require to share a PRNG that I control (say, noise generators for procedural textures), I cannot compose them if one of them uses the old and one of them uses the new version of the library.

• crates that expect to be used for interoperability are often split into smaller crates (like API and back-end, or low-level API and high-level API), so that they can evolve some parts without breaking others.

• sometimes breaking changes are technically breaking, but easy to upgrade (e.g. methods renamed, args reordered). In that case most users catch up quickly.

• in desperate cases, a new version can import its own old version and re-export old structs and interfaces that haven't changed, so they're compatible across major versions.

• proper sharing and composition should be done via traits, so that you can implement a trait for any number generator, not just one version of one implementation.

Maybe in the future we'll see more hacking of libraries (people managing to deliberately sneak exploits in) and in response stronger lockdowns on important library code.

I don't know how many hours I've spent battling Psych errors because of this very thing, but it's way too many. Calling the gem something, anything else, would solve the issue.

It's great that they're unbundling a lot of things, but there's still some serious friction between external and internalized versions of these gems.

For Ruby, EventMachine sub-universe is really in bad shape. EventMachine is creaky and old. Event-aware packages are in short supply and are usually woefully out of date, unmaintained.

But I think a lot of the value of a large standard library is that it makes it possible to write more programs without needing that first third-party dependency.

This is particularly good if you're using Python as a piece of glue inside something that isn't principally a Python project. It's easy to imagine a Python script doing a little bit of code generation in the build system of some larger project that wants to parse an XML file.

When you have no third-party dependencies, then adding the first one requires picking amongst trade offs and lots of work. A subset of choices include using virtualenv, using pip, using higher layer tools, copying the code to the project, using a Python distribution that includes them, writing code to avoid needing the first dependency ...

* You have to document to humans and to the computer which of the approaches is being used

* Compiled extensions are a pain

* You have to consider multiple platforms and operating systems

* You have to consider Python version compatibility (eg third party could support fewer Python versions than the current code base)

* And the version compatibility of the tools used to reference the dependency

* And a way of checking license compatibility

* The dependency may use different test, doc, type checking etc tools so they may have to be added to the project workflow too

* Its makes it harder for collaborators since there is more complexity than "install Python and you are done"

I stand by my claim that the first paragraph (adding another dependency) is way less work, than the rest which is adding the very first one.

[1] https://github.com/jazzband/pip-tools [2] https://docs.pipenv.org/en/latest/ [3] https://tox.readthedocs.io/en/latest/

I do think there's a lot of low-hanging fruit where Python could bake something in to auto-setup a virtualenv for a script entrypoint & have the developer just list the top-level dependencies & have the frozen dependency list also version controlled (+ if the virtualenv & frozen version-controlled dependency list disgaree rebuild virtualenv).

I used it to distribute dependencies to Yarn workers for PySpark applications and it worked flawlessly, even with crazy dependencies like tensorflow. I'm a really big fan of the project, it's well done.

https://github.com/pantsbuild/pex

[0] https://pypi.org/project/lxml/

It's also a nice API for dealing with XML.

I had enough trouble using it efficiently that I went and wrapped Boost property tree[0] and can happily churn out all sorts of data queries (including calling into python for the sorting function from the C++ lib) in almost no time.

I was taking daily(ish) updates of an rss feed and appending it to a master rss file but sorting was pretty slow using list comprehensions so now I convert it automagically to json and append it as is. No more list comprehensions either, just hand it a lambda and it outputs a sorted C++ iterator.

Though I probably should've just thrown the data into a database and learned SQL like a normal person...

[0] https://github.com/eponymous/python3-property_tree

If only that were the norm amongst long-tail python users. Heck I don't do it; I have the anaconda distribution installed on Windows and when I need to do a bit of data analysis hope I have the correct version of packages installed.

Making this core to the python workflow (bundling virtualenv? updating all docs to say "Set up a virtualenv first"?) is the first required change, before thinking about unbundling stdlib

And it is a totally miserable experience on Windows, every single time.

But, I do wonder whether it needs to grow. Python is in a stage where adoption of new std library features is inherently slow, not only in third-party libraries like twisted, but also in applications, even if they use newer Python versions.

What kills Python here is that it is most commonly bundled with the linux distribution or the OS (true also on mac). This reduces cycle times drastically. Compare this to newr language platforms that people like to install in newer versions on older platforms quite regularly.

Some recent additions would be fine additions at an early stage of language development, but surely not for Python.

It doesn't kill it, quite the contrary, rather makes it ubiquitous. If you want another version you just install virtualenv. It's the same with Perl. We use the Perl version shipped with the distribution (openSuSE) and deploy to that. It's older but it's stable and it works. On our dev environment (mac) we have the same version with all of the modules installed in plenv. We also chose a framework with as little deoendencies as possible (Mojolicious). It looks like it was a great choice.

So all you need is one dep that needs Guava version X with method M that is removed in version X+2 (say) and another dep that needs something new introduced in version X+2, and you have a Guava version conflict. That's, Guava releases are not backwards compatible due to removal of classes and methods.

You can sometimes fix this with a technology like shade or OSGi or whatever to allow private copies but it does not always work.

https://www.google.com/search?q=guava+nosuchmethoderror

The reasons are similar, it's a constant drag on core compiler development to need to support various batteries included that most core contributors aren't going to care about, so it's easier to tell people "use CPAN".

There was even talk of "distros" for the interpreter. Where the core bits would be similar to what Linux is, and all the batteries would be provide as collections of add-on packages.

Strangely enough these efforts seem to stop at OS distributors. They really seem to like to install just the one "compiler", and wouldn't stand for a project like Perl or Python telling them "we mean for you to distribute the core compiler plus these 100 packages, because that's what forms our 'language'". "Strangely" because you'd think they'd be the best positioned to make easy work of packaging up such a thing, and it shouldn't in principle make a difference if you need to install 100 RPMs / APTs by default.

When did you have to use cpan in a modern system? Compare that to how many times you had to use pip.

Now, if you use a crappy OS or distro (or god forbid, some container built by you have no idea who on top of nobody knows what) then yeah, you are bound to do the leg work yourself, but you will be doing that regardless of the language/subsystem you are trying to use in that case.

Not to mention that it is the only way to do things professionally. For example, if you must have a system that parses XML but for company policy is not allowed to have even the means of performing a network request. With python you either have both xml and an http library and whatever else included and you will either have to do a special package with a striped down python+xml only or get a corporate exception. While on other languages you can install only the xml parser component package and your code will run happily and be compliant with company policy.

Well yeah. A sizable amount of new software is still being written in Python. But when I use Perl software (besides my custom scripts), it's always stuff that's old enough that the distribution is carrying packages for it.

If you disagree, please name a significant new software written in Perl that was released in the last, say, 5 years.

I see your comment as the goal, not the problem.

...wouldn't the company policy involve removing the means of performing a network request from the computer, making the notional capabilities of the software irrelevant?

Python will let you drive network requests through the OS. It's just that you wouldn't normally want to.

That alone have probably paid of handsomely over the years considering all the XSS we patched, which could very well have been full network compromises.

I mean that a significant use people get out of Python and Perl is that they aren't bare-bones like say Scheme or Lua where the standard library is really spartan.

It allows you to write useful code that works on the lowest common denominator of "just OS Perl or Python". Whether that's some random version on whatever Linux distro, or *BSD or Solaris or whatever without needing to write your own getopt library or whatever.

Which is why the "let's ship a bare-bones compiler and have people use CPAN or PyPi" is contentious. In theory it shouldn't matter, and for a lot of shops who install hundreds of packages it doesn't, but it does for people who target stdlib-only, which is a big use-case. Particularly since the people who have that use-case are drawn to these languages.

But you don't have to ship just a bare-bones interpreter to deal with the problem of stdlib staleness, you just need the stdlib libraries to be updatable via package manager, you don't need to not ship a baseline version of them with the interpreter.

That doesn't deal with the bloat issue raised with relatively unused libraries, but if they are relatively unused because they aren't good rather than because the use case is uncommon, upgradability could solve that.

Of course, you don't solve compatibility for versions before the move to upgradable packages, but at the same time if you solve problems going forward you increase the incentive to upgrade.

Now not only do they need to ship a stable compiler+large-stdlib, but they can't even rely on there being a 1=1 version relationship between the two, instead it'll be many=many as users might use multiple library versions with multiple compiler versions.

Well, yeah, but you're not going to have much of a language at all if you optimize for quality of life of the language maintainer.

I've been using built-in environment isolation tools such as virtualenv for ages but have recently switched over to using miniconda for all things python. Among other things it has amazing support across all three major OS's, and I happen to be dealing with all three at any given time. Whether one uses miniconda, pipenv, virtualenv, or anything else like it, as far as I am concerned the days of ever using the system python are over. I will always create my own personal "distro" on the fly with full control over the python version and every add-on package.

You don’t have any Python scripts in your bin folder?

[0]: https://github.com/pyenv/pyenv

Not a million miles from the modularisation that Java has been going through.

As an aside, why doesn’t the Python standard library extend/replace features with code from successful packages like Requests? Tried it and it didn’t work? Too much bloat? Already got too much on the to-do list?

This may somewhat explain why lots of successful packages are not in the stdlib: putting them in effectively killed future development until a few years ago. But today that argument is inconsistently applied and I'm not sure if it's a rule worth keeping.

Pypi have thrown out the downloads counter—a huge misservice to coders. Like I got all day to figure out the best libs for ten different features which I only need in passing, so my primary concern is to not pick complete garbage.

So, my solution to that now is to look up Github pages for the libs and choose the one with most stars. As much as I dislike Github for its occasional typical proprietary behavior, Gitlab doesn't help in this case.

A better approach might be to add a core of basic requests-like features built from the stdlib’s existing resources. That would be beneficial to many users and if they need more then there’s always Requests.

To the best of my knowledge, minor version updates in Python 3 have been entirely backwards so far and shouldn't have broken any library code. And as for incompatible changes sick as the transition to 3, then these of course had interface changes in the standard library.

I'm not entirely sure what you're looking for, but have you tried https://www.enthought.com/product/enthought-deployment-manag... ?

Edit: example

    $ edm envs create tester36 --version 3.6
    $ edm shell -e tester36
    (tester36) $ edm install ipython matplotlib pyqt

> As an aside, why doesn’t the Python standard library extend/replace features with code from successful packages like Requests?

It is possible (ie. asyncio was separate package). It is slow process though.

https://python.libhunt.com/ Not exactly curation but it does have rated libs for lots of categories.

That would be my guess. I would also add that getting through the process of adding a third-party set of modules to the standard library can take quite a while.

Asyncio is in the stdlib so that we have an official lib and API. The main benefit is that most people now, when looking for async, are not wondering about twisted or gevent or tornado. Most just go asyncio. Most dev efforts go to asyncio. It's the end of the great async war. Is it perfect ? No. And I don't care. It's one thing less to worry about. For those who know what they are doing, you can still choose and pip install twisted, but most people don't, and that's solved. Before that, just choosing the lib was a nighmare, as basically it's a definitive call. Out it on pypi, even with a "stdlib" tag, we go back to the 200X era. And it was not fun.

And the goal for having things like xml/sqlite/ssl without installing anything makes python very useful in a load of situations where you can't install stuff. Sometime you are offline. Sometime you are in a restricted env. Sometime you are not on your machine. Sometime your security protocol is hell. Don't assume people use Python as we do, from our comfortable dev laptop driven by the knowledge of our craft. Python is used in banks, by scientists, in schools, by kids, by poor people in the third world, by geographers and pentesters. The python user base is incredibly diverse, it's why it's so popular: it fits a lot of use cases.

So I see the benefit of having a side version of official modules we can pip install that can move faster. I see the benefit of cleaning the stdlib of old stuff, like the wave module, Template or @static.

But I'm glad I don't have anything to install to generate a uuid or unzip stuff. I'm glad I don't have to worry about twisted anymore (depiste that I did write a book on the topic !).

Also, pip install is NOT simple when you learn the language. I have to spend some time in the classroom, even with adult professionals, to explain the various subtleties of site-packages, import path, py -x on windows, python-pip on linux, -m, virtualenv, header files, etc. before my students become autonomous with it. Without a teachers, this turn into months of bad practices and frustrations.

You'd have to fix that first, way, way before moving stuff to pypi. I do think it should be high priority actually: it affects way more than pip.

If you're in the flow and trying to hack together something, the last thing you need is to lose all momentum to pick a date time library. I've had this issue tons of times with Node and Rust, where I'm not up to date with the current meta and my 30 minute hack job is interrupted 5 minutes in by having to google which library should I use to do an HTTP request. (I've actually lost interest in whatever I was doing a few times because of this.)

Python's stdlib is nobody's favourite, but when you start to get to its limits, you're probably past your flow state, you've written most of the logic and you can spend some time to replace http.client with requests because the latter is much better.

On a tangent note, I've been trying to find another scripting language to replace Python because I'm not a fan of it anymore (I won't get into it right now), and considering what I just wrote, there's not much that can replace it, as most languages have a bare-bones standard library and if you're not up to date with the current best library to do X, you'll never achieve great productivity.

Have you considered ruby?

Wait... what ? No way ! Some of us do you use Python to process wav files. If anything, I'd like this module to be updated, not removed.

It was pitched first as a common low level async loop for other applications like Twisted and Tornado.

Then people started using it directly and the keywords were added.

It's great for the people who think the way asyncio does, others are now forced to use it. I find all of Twisted, Go and Jane Street's Async easier to use.

Perhaps Python is just the wrong language for me.

Explain? I’ve hated the slow adoption of 3.x from 2.x, and generally how terrible it is to have apps that are 2.x on your 3.x system, and would like to know more about how that happened.

Perl has a much better set of modules that extend standard functionality, which considering how much flack Perl gets for being hard to read, is rather funny. Rather than every new feature being its own independent project, most of the useful modules inherit a parent and follow the same convention, leading to very simple and easy to use extensions. And Perl Core isn't all that great, but it does have some batteries included, and everything else is extended easily and in a more standard manner by CPAN.

Sure, I've found some gems on CPAN, but, having worked on both Perl, Python, and Java at reasonable scale for awhile, I cannot understand all the praise CPAN gets. It's the worst-quality scripting language package ecosystem out there. Even NPM does a better job, and some things about NPM are awful. CPAN might have been the first/only/best package manager for a get-shit-done scripting language at some point, but not any more.

Separately, I agree about modules which extend language functionality (e.g. class systems, async programming, runtime typing) specifically. Perl does pretty well in that area. While many of those language-extension modules really don't play well with any other metaprogramming tools being installed in the project, I don't imagine that any alternatives in other languages do, either. My main beef above is with "simple" (read: not pervasive semantics changes) modules like IPC utilities, HTTP clients, or loggers that don't know how to stay in their lanes.

Search engines are a "cool" technology that have become the de facto way to find what you're looking for. But if there's a lot of content related to what you're looking for, they can suck.

Go to PyPI and search for "semantic version". 10,000+ projects for "semantic version" found. As you go through page after page of different modules related to versioning, the one module you won't find immediately is Versio (https://pypi.org/project/Versio/), a well-documented and useful module which I ended up using. I have no idea how I found this module, but it certainly wasn't from PyPI's search engine.

Now go to CPAN (really metacpan) and search for "semantic version". Yes, you're still looking at thousands of results - but wait! There are only two modules here that look useful: Version::Dotted::Semantic, and SemVer. And the description comes straight from the docs' README, rather than being a short uninformative blurb. The first module, Version::Dotted::Semantic, is inheriting a separate module, Version::Dotted, and adding some extra functionality. Not only does the search page give more information about the module, but the hierarchy makes it easier to find (and later extend) useful modules in an intuitive way. Since the base module's functionality is boring, generic, and simple, it's less likely that people will make 20 different versions of it, so it'll be reused more often and thus remain stable for a long time.

A lot of CPAN's module names have sprawled over time and gotten less useful, but there's still a general convention that you name your module as a hierarchy of what it does (even if it's kind of verbose) and make small, reusable modules, rather than giant modules that are hard to extend. Not all modules measure up to this standard, and there's definitely room to improve, but I think Python modules could benefit greatly from a system like this.

As far as curation goes, PyPI is often filled with cruft. While searching for Jenkins packages, you will come across lots of entries like this: https://pypi.org/project/jenkins2api/. The homepage leads to a GitHub 404, it's only ever had one release, and it has no documentation. This project should probably not have been listed on the main search page, or at least sorted well down the list by default with intelligent filters and marked accordingly. (The "date last updated" and "trending" sorting just results in having virtually no Jenkins-related modules in the results at all)

I’m not sure if Python’s ideal solution is to reduce stdlib and have endorsed packages in PyPI, but it would be an improvement over the current process.

> Van Rossum argued instead that if the Twisted team wants the ecosystem to evolve, they should stop supporting older Python versions and force users to upgrade. Brown acknowledged this point, but said half of Twisted users are still on Python 2 and it is difficult to abandon them. The debate at this point became personal for Van Rossum, and he left angrily.

The end of life date is already set: January 1, 2020.

It's open source so I don't know what you're looking for in terms of a formal handover. Yes, I bet Red Hat and others will continue to maintain their own versions past that date.

To ensure things move along: pip has been printing highly-visible "python 2.7 will deprecate soon" warnings for a couple months or so now.

In Python's defense, this was not obvious at the time; my understanding is Rust came to this approach by looking at the experience of Python and other languages. When Python's standard library was first being written, there were no easy package managers for any language, and the normal thing to do for installing dependencies in e.g. C was to grab random tarballs and figure out how to build and deploy them yourself. So avoiding that process made perfect sense.

Obviously it doesn't apply to everyone, and it certainly doesn't apply to most startups or open source developers, but I spent most of the last 20 years working in environments where you have to get permission for every third party library you bring on to the network. Many networks were essentially "airgapped", so it's not like you could just ignore the rules. The bureaucratic process alone meant that we preferred large bundles like Anaconda or Qt. Trying to use Cargo as it is typically used and documented would be a complete non-starter.

Situations like this will really make you appreciate "batteries included". I think this particular issue is fairly revealing of the attitudes common among programmers of different languages. I think it's a good thing to be skeptical of a program pulling in a bunch of standard libraries over the Internet. It worries me when I find something on Github I want to try and I can't download and compile it without it pulling in 30 or 100 other libraries that I haven't looked at or decided to trust come along for the ride. I don't like that way of doing software, and unfortunately it's the norm in node and starting to become a norm in Rust as well. Real security fails have been caused this way in node's case at least.

Even in cases when Python programs depend on external libraries, I usually don't need to use pip for anything because Python programs will pull in dependencies provided by your distribution just fine. (My distribution doesn't even have any Rust libraries, so even if dynamic linking is possible in Rust not many people are shipping software that way.)

"Download by default" is a worse way of doing things, and it makes me sad to see newer languages like Go and Rust embracing it.

We do have an internal PyPI mirror (with devpi) and we point `pip` at that, and it works pretty well.

(Pointing at an internal mirror is now stable. Setting up that mirror is the hard part.)

So Rust can get away more easily without having support for things like command line parsing in the standard library, because it isn't really trying to support situations where it would be inconvenient to give your program its own project directory and Cargo.toml and all.

If it really matters, then do it right (in whatever language makes the most sense).

Wasn't perl's CPAN developed around that time period (mid '90s) ?

Yes it does. Perhaps you're thinking of browser JS, which does not.

Edit: based off of all the replies below, everyone understands the validity of what I'm trying to say, but also have fortunately pointed out my admittedly grevious error of not knowing you can just import hashmap from stdlib. The extra step is pretty minimal and not a problem. I'm hoping this is covered in the Rust book.

https://doc.rust-lang.org/std/collections/struct.HashMap.htm... https://doc.rust-lang.org/std/collections/struct.BTreeMap.ht...

    use std::iter::FromIterator;
    BTreeMap::from_iter(hash_map.into_iter())

  for (k,v) in &src
    { dst.insert(k,v); }

?

That being said the point still stands, I remember specifically feeling it at the lack of included regex. I overall prefer the slim std lib of rust over the massive python though. Especially with the community being pretty good about nominating de facto standard packages like for regex.

https://doc.rust-lang.org/std/collections/struct.HashMap.htm...

I didn't know it was included in stdlib (I really thought it wasn't) and feel idiotic now. It is still odd (having a primarily scripting background and not coming from the systems side) that I have to include what seems to be essentially an import statement at the top. I guess it is a lot more efficient that way though. Thanks for pointing out my error!

There are other new stuff that might be confusing to beginners (like our mighty walrus operator), but those two examples have been there for effectively forever.

I just had the feeling there's more "stuff" that you need to know.

This is actually an interesting paradox to be in, and one that Linus Torvalds recently commented on. His focus, like Guido’s, is the user and even fixing a bug can break the user.

https://lkml.org/lkml/2018/8/3/621

That's how you get an inconsistent mess that never evolves. There's something called semver, increase the version number and do the fix / refactors / radical redesign / whatever. People will see that you've went from version 1.0 to 87.3 in one year and they may choose not to use your thing because you're moving too fast for them, but that's life...

Amber is nothing short of an open source hero, having brought Twisted, one of the best open source projects in the world, to new heights. Her insights are as important as anyone in the python community, and after six consecutive PyCons sprinting at the Twisted table (including literally in a chair with Amber to my left and Glyph to my right earlier this month), I consider Amber's voice to be one of the truest and clearest among the leadership of the language into the future.

Amber and Guido are both beautiful human beings.

In the dispute that is the topic of this blog post, Amber is basically totally right. Moreover, the distinction has less to do with any kind of nagging python 2 holdover than this article suggests. The standard lib's role as a place where code goes to die is a view that is widely held and accurate for many cases.

The following question went unanswered during the Steering Council Q&A:

"Every feature request has a constituency of people who want it. Is there a constituency for conservatism and minimalism?"

...and that's really what this whole thing is about.

I agree with a lot of her arguments. I never understood why tkinter was included (I could see why it might have been added years ago, but before Python3 came along it felt obsolete and unused). I've had to support old versions of Python along with their bugs, and it sucks.

But languages are about choices. Python's syntax and decision to use whitespace is a strong choice--so is their stdlib. I've been on projects that saw performance problems with etree, moved to lxml, then had to go back to etree because of missing features. I've spent a lot of time looking over arrow, dateutil, and moment because datetime seemed inadequate. But I like that there is a thoughtful default that serves many needs. A lot of these examples, like requests, are built right on top of stdlibs--so both would be needed even if it was shipped along with stdlib. I'm ok if code goes there to die because I would hope that due diligence was taken when it was included.

I kind of agree with Guido for a lot of this. I can't wait to move from Python 2 and have looked a lot of the new stdlib and looked for backports. I've looked at twisted and alternatives and I'm so happy something is built in even if its far from perfect.

When it comes to constructive criticism, I think Amber did a good job with her criticism but can do better at the constructive front. Her problem statement was spot on and I agree that the direction she proposed is a good one.

However, to separate the standard library from the core is probably even more dramatic than the Python 2 to 3 migration. Is that what the community can afford at the moment? What's needed to make the transition? What's the opportunity costs? i.e. what other developments we can do for a bigger impact? What are the pros and cons?

My problem is on the like part.

Where shall we draw the line and how do we decide? To me this is a far more interesting discussion. (Maybe it has happened. I don’t go to many conferences these days so I might be missing something here. )

She mentioned http.client vs requests, datetime vs. moments etc, which are also quite correct to me. How about the cgilibs? Or pickle? Or the collections? Or unittest? Stay or go?

Lastly, the title of the talk can be tempered a bit. No? We all know what a leaking battery mean right? Toxic.

Why does there need to be a line? As long as the package manager is part of the core distribution (even if it is itself an upgradable package) why not moving everything into packages, even if some are maintained by the core team and have the stable version at time of distribution release included with the core distribution—but perhaps installed only on demand?

pip install unitest? Oops I spelled it wrong wonder what I just installed?

“have the stable version at time of distribution release included with the core distribution”

Ruby, for instance has both “default” and “bundled” gems with the core distribution.

https://stdgems.org/

Kidding aside, most of the problem with the fake libraries could be solved if pypi namespace them per author, like GitHub does for repos.

Come on. Python has had the 'batteries' metaphor for decades and this is a straightforward and obvious play on it. You're bringing 'toxic' into this which I suppose is technically and biochemically correct (both, surely, the best kinds of correct) but has far, far more negative connotations than the title warrants. You're having to work really hard to make a generic thing sound dreadful.

To be honest, I think she might be biased here given that she maintains a competing package.

I hope that you take time to reconsider this view.

We're not talking about competition in the same sense as in a capitalist system, between two companies.

Prior to asyncio, Twisted maintained the only viable flow control for serious asynchrony in python. Put another way, python had no standard flow control for serious asynchronous abstractions in the standard language (and standard library) before asyncio (and `await/async def`, etc) landed.

Nobody is saying that the syntactical changes to python belong in a separate package on PyPI (cue Gary Bernhardt's Pretzel Colon). These things are fine.

But `asyncio.Future`? Yeah, I see a very reasonable argument for that stuff (ie, the asyncio namespace) being in a separate package.

But OK - looking again at the "competing package" narrative: now that these things have landed, Twisted has done an amazing job of using them alongside all the other tooling that Twisted also provides, most notably its test infrastructure.

Amber doesn't stand to personally benefit from asyncio failing. To the contrary, having the flow control taken care of so that Twisted doesn't have to be its sole brainparent gives her much less free work to feel obligated to do.

Could you provide a reference/explanation for this? Thanks in advance!

[0] https://www.destroyallsoftware.com/talks/wat

[1] https://www.destroyallsoftware.com/talks/the-birth-and-death...

[2] https://technology.customink.com/blog/2015/06/08/ruby-pretze...

And it wasn't the only thing at the language summit that proposed expanding that approach: there was a discussion of carrying time zone updates in the same way, by shipping something with the interpreter that works but allowing updates from PyPI. http://pyfound.blogspot.com/2019/05/paul-ganssle-time-zones-...

So I think the development community / target audience at the language summit already understands the pros and cons of the suggested approach and the technical route to get there. (For an end user, my guess is the experience will be that anything in the Python 3.x standard library today will still be in the Python 3.x standard library, but you'll have to `pip install` a newer version if you want more features, and you get the benefit of being able to `pip install` something from the standard library where you previously couldn't.)

What do we know? Software configuration management is a heinous problem.

Python gets it more correct than most.

Let us rejoice, be patient, and respect everyone's good-faith efforts.

Amber wants to remove asyncio from the standard library. That's minimalism but not conservatism.

Also note that one way of interpreting her proposal is "generalizing the 'ensurepip' model" to keep things already in the standard library in the standard library, but move feature development externally and make it easy to upgrade packages in the standard library. (Then Twisted, which is also installed externally, can simply depend on a fixed version of a standard-library package.)

Perhaps, but Guido's response was shitty here (and purposefully refusing to get the point), whether one think Amber was right or not.

I deal constantly with the shitshot that is Python packaging (and lack-luster default packages) and I happen to think she is 100% on the spot.

Did asyncio module feel bloated? It certainly did. It seems like every module from subprocess to networking to io is crammed into it.

On the other hand, did it get the job done without resorting to any packages or threading? Yep, and that is pretty powerful and rare.

I think bloatedness of the stdlib isn't actually a practical problem. It's just an inelegance that you kind of have to learn to tolerate.

I maintain a Python 2 & 3 compatible project that has no external dependencies.

In some cases of common OS-induced pain, I'd say "do whatever 3 does" in 2, since that'll make migration easier in the long run. (But I understand that can be hard, and I think my responses to your examples below even demonstrate that to be hard.)

To your specific pain points:

> Unicode stdio?

Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2.

> Unicode file paths

This is going to be a mess in any language, because file paths really aren't text. On nix, they're byte strings that don't have nuls in them. Hopefully* they're encoded according to LANG, and hopefully LANG is a UTF-8 variant, but it isn't required, and it isn't required that two users on the same system use compatible LANGs, so you get a Tower of Babel. I really wish OSs would just start enforcing a. Unicode filenames, and b. no newlines in filenames; those two alone would make life so much easier.

Hopefully you've seen os.fsencode / fsdecode, but alas those aren't in 2, so I'm not sure they really help you. Often one is not really munging paths that much, and can just pass through whatever value/type you get, but it does happen, of course. (E.g., adding or removing extensions)

> Unicode sys.argv

This is also a pain point, since again, the underlying type in nix is a byte string without nuls. I'd hope it decodes w/ the LANG encoding, but since the user could easily tab-complete a filename, fsencode/decode might be more appropriate. I think I'd say "do whatever 3 does".

1 Jan 2020 is nearly here. Forget about 2 / assume UTF-8 in 2 and don't support anything else?

> I feel like Python 3 completely wrecked strings instead of making them better.*

A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.

> Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2.

Does io handle stdio? I was referring to standard input and standard error/output. How do you read/write Unicode in a cross-2/3 way from/to standard input/output/error without adding your own translation layer?

>> Unicode file paths

> This is going to be a mess in any language, because file paths really aren't text.

Sorry I need to be more clear. That general mess is not the aspect of it I was referring to. I'm specifically referring to a 2/3 compatibility mess.

I meant that, for example, to have any semblance of Unicode handling, in Python 2 you do os.listdir(u"."), whereas in Python 3 you do os.listdir(b"."). I know how to handle it in both Python 2 and Python 3 in a way that's Good Enough (TM), but how do I even get that with cross-compatible code? I'd need to write a translation layer of sorts for every single I/O function I might use.

>> Unicode sys.argv

> I think I'd say "do whatever 3 does".

Hmm okay thanks, I'll need to try to see what the implications are again. I think the problems I recalled from this may have been just been a result of the other issues, not sure.

I can't "forget about 2" though, it's still on Ubuntu LTS systems and there are still packages in 2 that haven't been ported to 3.

>> I feel like Python 3 completely wrecked strings instead of making them better.

> A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.

Again, I think this is "cleaner" in theory, not in practice. What happened to the string_escape/unicode_escape nonsense I pointed out with the new system? Any rebuttal to that one? ;-)

No. No on just so many levels.

It finally fixed it by introducing two (or three if you count bytearray) types for completely different semantics of data.

Python 2 was just a mess whenever you left ascii.

But looking solely at python3 , everything byte and unicode related just got so much easier and better testable and with better error messages at better points in your code than in python 2.

So claiming python 3 broke stuff just not what the case is.

Python 2 had a deeply flawed, boundryless view of unicode vs bytes and python 3 fixed that.

What is your reason for using this obscure encoding anyway?

Yes and my entire point is that this makes zero sense when we're talking about escaping.

> What is your reason for using this obscure encoding anyway?

Seriously?

The encoding `unicode_escape` is not about escaping unicode characters. It's about python source code. It's defined as:

> Encoding suitable as the contents of a Unicode literal in ASCII-encoded Python source code, except that quotes are not escaped.

It makes absolutely no sense to have escaped unicode characters as actual unicode string. If you really need that, a version that also works in python 2 would be:

     u'hellö'.encode('unicode_escape').decode('ascii')

Absolutely no sense? So is a basic Python expression evaluator like this complete nonsense to you? #!/usr/bin/env python3

  try:  # Python 2
   from Tkinter import Tk, Entry, END
   import tkMessageBox as messagebox
  except ImportError:  # Python 3
   from tkinter import Tk, Entry, END, messagebox

  import ast
  def my_eval(s):
   # assume I've implemented this functionality manually...
   return ast.literal_eval(s)

  root = Tk()
  e = Entry(root)
  e.insert(END, '"Hell\\xc3\\xb6"')  # Assume the user typed has typed in a Python expression, not me
  e.pack()
  root.bind('<Return>', lambda evt: messagebox.showinfo("Result", repr(my_eval(e.get()))))
  root.mainloop()

Note that that wasn't even my choice! That was the choice of the built-in Python GUI toolkit... distributed by the same folks who decided this string/bytes overhaul was a brilliant idea...

After A TON OF PAIN I decided to put the time in to watch the talk "Pragmatic Unicode, or, How do I stop the pain?"[0] by Ned Batchelder and it all just clicked. Now, I make unicode sandwiches like a boss.

[0] https://youtu.be/sgHbC6udIqc

Does writing python 2 code stop being 'worth it' at some point? If so, where is that point? It sort of sounds like you're already there. There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no?

I'm not a professional python dev, so it doesn't impact me as much as some of my colleagues, but 'backwards compatible' issues crop up in other platforms/langauges as well. Wordpress might be the biggest example in PHP. They've kept a minimum target language which is far behing 'current' or even 'currently supported', and it's been a catch 22. Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4. WP 5 was, alas, a missed opportunity to target PHP7 as a minimum. :/

How do you know I'm there? If nothing else I still have my own previous Python 2 code that I've spent time on and that's useful to me! Why would I just throw them all away or waste massive amounts of time rewriting them into Python 3? Is my goal supposed to be to please the masses here?

Honestly that's sufficient reason already. But if you want a more "standard"/politically-correct response: Python 2 is still being supported, Ubuntu 14.04 LTS literally only just reached EOL last month and can be found in the wild, Python 3 support is lagging in a lot of places (e.g. PyPy is still on 3.5, although I'm just using it as an example; I don't use it much), and I still come across Python 2-based tools and packages once in a while.

So I guess the answer to your question is it "stops being worth it" when I stop coming across situations where I'd regret abandoning Python 2.

> There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no?

"Because" is an odd way to put it... it's true that if they didn't support P2 then people would use P2 less, but people use it because it benefits them, not "because it's supported". I mean, you're also alive "at least in part because" of all the hoops people jump through to grow and bring food within your reach, but I doubt your conclusion is that this should stop being the case...

> They've kept a minimum target language which is far behind 'current' or even 'currently supported', and it's been a catch 22.

What I don't understand is why are people supposed to keep abandoning good software just because someone made something shiny and declared it "unsupported"? I hate this "you have to like my updates... or I will force you" attitude that every organization seems to have nowadays. People are trying to solve their own problems, not please the leader they're following.

> Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4.

I mean, this isn't even the same situation? I'm not targeting Python 2 or introducing a dependency on it generally. I try hard to keep my code both Python 3 and Python 2 compatible. So the decision as to whether to move on or not is left to the client (which is often myself) and there's no obstacle either way.

Part of the argument for using much of the stuff out there is because it's "supported" (commercial, community, etc). Without some degree of support, things dwindle/die. The 'support' for language stuff may simply be security patches. If/when those stop being provided, you're starting to do a disservice to people to continue to push that language, even if the core functionality you need is still working and perhaps even unaffected.

> I'm not targeting Python 2 ... I try hard to keep my code both Python 3 and Python 2 compatible.

Given that there are breaking changes between the two, I can't read this any other way that that you're 'targeting' Python 2. If there's a way to do something in 2, and it's not working that way any more in 3, and you're making code so that it runs in both (weird syntax, translation layers, etc) then... you're targeting 3. The same way that if you use 3 syntax, you're 'targeting' 3. That's just how it is.

> but people use it because it benefits them, not "because it's supported".

The benefit is that they don't have to go through whatever hoops it takes to upgrade to have the latest versions. In many cases the later version may not add much directly, but 'community support' and 'security' are two ... emergent properties of a critical mass adopting new versions and dropping support for old versions.

> How do you know I'm there?

Wild-ass guess from the gist of all your earlier posts in this thread?

> But if you want a more "standard"/politically-correct response: Python 2 is still being supported,

I'd say that's partially 'under duress'. My reading of the situation years ago was that the Python community - at least the leadership - wanted to EOL P2 on Jan 1, 2015. There was a gigantic stink/pushback, and support was extended for another 5 years. I will grant that the v3 switchover was pretty bad - I'm not a pro user, but had some colleagues that dealt with a lot, and certainly, 8+ years ago, it was pretty hard to just 'switch' running systems. Even developing new stuff from scratch - there were a lot of 'only works in 2' libraries that were entrenched standards without clear update paths. However, rather than doubling down and making 3 a more attractive proposition (work on upgrade path, faster language, better docs, whatever), the agreement was to support P2 for another 5 years. That's an eternity in the tech world. You could argue it shouldn't be, but it is. Yet, from python colleagues - those who've stayed - the upgrade story isn't significantly better. I'm taking their word for it, but based on your posts, it doesn't seem to be either.

fwiw, six can easily be vendored into a project to avoid the technical external dependency. that is how we manage it for kafka-python.

there are a few modules that are simply at different locations but have the same API

  if PY3:
    from http import client as httplib
  else:
    import httplib

  PY3 = sys.version_info >= (3, 0)
  PY2 = sys.version_info < (3, 0)
  
  PY26 = sys.version_info >= (2, 6) and sys.version_info < (2, 7)

  isinstance(<maybe_string>, basestring if PY2 else str)

  # Python 2.6 doesn't properly UTF-8 encode syslog messages, so it needs
  # to be performed in a custom formatter.
  formatter_class = UnicodeLoggingFormatter if PY26 else logging.Formatter

As the maintainer of WebOb/Pyramid/Waitress we have a compat module that contains all of the changes/renames/functions to help with the Python2/3 compatibility and all tests run across both platforms.

Are the functions borrowed heavily from six? Yes, but we don't need to vendor all of six.