Most of these applied fields are using Python and R as no more than data gathering tools and fancy calculators. something for which the benefits of other languages are just not justified.

The absolute beauty of Python for what I do is that I can write code and hand it off to a first year with a semester of coding experience. Even if they couldn’t write it themselves, they can still understand what it does after a bit of study. Additionally, I can hand it off to 75 year old professors who still sends Fax memos to the federal reserve and they’ll achieve a degree of comprehension.

For these reasons, Python, although not perfect, has been so incredibly useful.

You’ll find everyone from philosophy PhDs to Biologists to Journalists who use pandas because its so easy to learn it and work with it. It’s amazing how you can become productive in python/pandas without any experience or even basic understanding of programming because of how accessible jupyter, colab and blogs/docs on pandas are.

The other thing people don’t talk about is that a lot of these organizations can hire a CS student part time or a full time software engineer/data engineer/data scientist who can optimize their scripts once they are written. Pretty much any software engineer can read and debug python code without needing to learn python. So for example, I know some engineers working in genomics who have turned biologist-written scripts that take several days to run in python into scripts that take hours or minutes to run by doing basic optimizations like removing quadratic algorithms from the script or applying pyspark or dask to add parallelism.

The fact that python can be used as a bridge between technical and non-technical people is amazing and I think it has provided a better bridge between these groups than SQL was ever able to provide.

1) Languages like Python / R / Julia / etc. + SQL

2) PowerBI, Tableau, or similar tools

3) Excel

The number of users of those tools will be the inverse, with Excel being number 1.

If you're competent using the "stack" above, you could probably work as an analyst anywhere - given that you can pick up domain knowledge.

And I wouldn't be surprised if a large number of python programmers would say they're not programming, it's just scripting.

It’s still my choice for quick and non-graphical analysis when I’m on a remote.

Today honestly most tools are pretty capable, pandas is a great choice and if you have really high volumes of data you might try koalas (spark) or polars.

Honestly the biggest design considerations for data science today are things things external to your project: what do you and others on your team know, what tools does your company already have setup, what volume of data are you processing, what are your SLAs, who or what else needs to run this script/workflow, what softwares do you need to integrate with, how often does it need to be processed, how are you going to assure the quality of your data and what tools are you using for reporting?

I tend to use pandas and SQLite for most use cases cause I can cook up a script in 2 hours and be done, I just code it interactively in a notebook and most people are able to work on a pandas or SQLite script productively if it needs to be maintained even if they don't know python. If its a large volume of data or a rapid schedule (minutes, seconds) or tight SLAs on quality or processing time, then I start to consider whether pyspark, Apache beam, dask or bigquery might be a good fit.

So it really just depends but for most people who are processing < 100 GB on a 1+ day schedule or ad hoc I would recommend just using pandas or tidyverse in R and getting really good at writing those scripts fast. Today you’ll get the most mileage out of those two tools.

When writing code that will be used by a particular sort of user base, the code should be written in whatever way best suits that user base. If your users are academics, researchers, journalists, etc. -- yes, avoid anything with complex or obscure semantics like perl or bash.

But if your code is going to be used by programmers or people who are already comfortable with perl/bash/whatever, those tools may be just the ticket.

They aren't technically "spaghetti", but they are technically impenetrable.

I argue that one-liners like that aren't good for anybody, dev or otherwise.

If you use the python repl a lot and haven't heard of it, ptpython is worth checking out as a repl replacement. I find it to be much more ergonomic.

https://xon.sh/

No need to fire up a python repl.

This piqued my curiosity. I've worked with Python on and off for the last ~20 years, and while I'm not a fanboy or apologist, and use other tools when appropriate, there's also a reason it remains in my toolbox and sees regular use while many other tools have come/gone/been replaced by something better.

Can you share an example scenario where it's a Monkey's Paw? My suspicion is that this is more of an org issue than a tech issue?

People work around these issues with various kludges like virtual environments, Docker (just ship the whole system!), and half a dozen different package managers, each with their own manifest format. But this is a problem that simply doesn't exist in Go, JavaScript, Rust, and others.

For code that never needs anything except the standard library, or for a script that never needs to be maintained or run on a different machine, Python is fine. Maybe even nice. But I've watched my coworkers waste so many hundreds of developer-hours just trying to wrangle their Python services into running locally, managing virtual environments, keeping them from trampling on each other's global dependencies, following setup docs that don't work consistently, and fixing deployments that fail every other week because the house is built on sand.

Virtualenvs, and requirements are a thing in Python for ages.

I’ve used tons of languages and while not the best, Python dependency management and project isolation is decent. IMO certainly better than JavaScript.

In contrast to Javascript/Node.js/NPM/Yarn/whatever-you-want-to-call-server-js, which maintains a local folder with dependencies for your project, instead of installing everything globally by default.

Heck, a virtual env is literally a bundled python version with the path variables overriden so that the global folder is actually a project folder, basically tricking Python into doing things The Correct Way.

[0]: https://docs.python.org/3/library/venv.html

I feel that it has recently - like many really mature platforms - become very much like the elephant from that old apocryphal story [0]. It is being used for many different purposes, with very different requirements and needs, with users being so focused on their own use that anything outside that is considered "bloat" and "waste".

[0] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant

fp languages like

ocaml

haskell

racket

common lisp

erlang

elixir

or rust/golang

but even if the code is single threaded and not hampered by GIL limitations python tends to be super slow imho; also debugging dynamic python and imperative stateful python after a certain code base size >10k LOC gets extremely painful

When we really want or need performance we rewrite the slow part in C++ and use pybind to call into it. For all the real implementations that run in a soft real time system, everything is done in C++ or C depending on the ecosystem.

for any meaningful scale you are better served by basic FP hygiene as evidenced in

haskell

elixir

CL/racket

or rust/golang

  (loop for x across numbers
        when (evenp x)
          do (setf result (+ result x)))

Nobody cares about this that much. Even a straight up software developer in python doesn't care. The interpreter is so slow that most optimization tricks are irrelevant to the overall bottleneck. Really optimizing python involves the FFI and using C or C++, which is a whole different ball game.

For the average python developer (not a data scientist) most frameworks have already done this for you.

By the time a python programmer has «graduated» to learning a second language, exponential growth has created a bunch of new python programmers, most of which don’t consider themselves programmers.

There are more non-programmers in this world, and they don’t care - or know about - concurrency, memory efficiency, L2 cache misses due to pointer chasing. These people all use python. This seems to be a perspective missing from most hackernews discussions, where people work on high performance Big corp big data web scale systems.

What worries me, though, is that the features that make Python quite good at prototyping make it rather bad at auditing for safety and security. And we live in a world in which production code is prototyping code, which means that Python code that should have remained a quick experiment – and more often than not, written by people who are not that good at Python or don't care about code quality – ends up powering safety/security-critical infrastructures. Cue in the thousands of developer-hours debugging or attempting to scale code that is hostile to the task.

I would claim that the same applies to JavaScript/Node, btw.

Immutability would be the default, but mutability would be allowed, marked in some concise way so that it was easy to calculate things using imperative-style loops. Pervasive use of immutable instances would make it impossible for libraries to rely on mutating objects a la SQLAlchemy.

The language would be statically type-checked, with optional type annotations and magic support for duck typing (magic because I don't know how that would work.) The type system would prioritize helpful, legible feedback, and it would not support powerful type-level programming, to keep the ecosystem accessible to beginners.

It would still have a REPL, but not everything allowed in the REPL would be allowed when running code from a file.

There would be a strong module system that deterred libraries from relying on global state.

Support for at least one fairly accessible concurrency paradigm would be built in.

I suspect that the error system would be exception-based, so that beginners and busy people could write happy path code without being nagged to handle error values and without worrying that errors could be invisibly suppressed, but there might be another way.

If someone who's not familiar with programming runs into an error like "why can't I change the value of X" that might take them multiple hours to figure out, or they may never figure it out. Even if the error message is clear, total beginners often just don't know how to read them and use them.

They provide longer term advantages once your program becomes larger but the short term advantages are more important as a scripting language imo

I don't think mutability by default is necessary for beginners. They just need obvious ways of getting things done. There are two places beginners use mutability a lot. The first is gradual transformation of a value:

    line = "The best of times, the worst "
    line = line.trim()
    line = line[:line.find(' ')]

    word_count = 0
    for line in lines():
        word_count += num_words(line)

    word_counts = [num_words(line) for line in lines]
    # or word_counts = map(num_words, line)
    word_count = sum(word_counts)

    word_count @= 0
    for line in lines():
        word_count @= word_count + num_words(line)

It's a common claim that immutability only pays off in larger programs, but I think the mental tax of mutability starts pretty immediately for beginners. We're just used to it. Consider this example:

    dog_name = Name(first='Rusty', last='Licks')
    dog1.name = favorite_name
    dog_name.last = 'Barksalot'
    dog2.name = favorite_name
    print(dog1.name) # It's not Rusty Licks!

Julia is both dynamic and fast. It doesn’t solve all issues but uniquely solves the problem of needing 2 languages if you want flexibility and performance.

Coupled with the dynamic typing and mutability by default, it guarantees Python programs won't scale, relegating the language to the role of a scratchpad for rough drafts and one off scripts, a toy beginner's language.

None of those applications are toys. I have no idea where your misperception is coming from.

All sorts of problems would simply have not existed if the proper language was used from the beginning, as opposed to the one where anyone can hack most easily.

It’s already what you get with python and mypy. Using typing.Protocol or Unions.

However, I hope that these C applications are written by people who are really good at C. I know that some of these Python applications are written by people who discovered the language as they deployed into production.

If the masterful C developer still insists on using a language that has so many footguns and a weird culture of developers pretending that they’re more capable than they are, then their C mastery could very well’ve not been worth much against someone throwing something together in Python, which will at the very least immediately bypass the vast majority of vulnerabilities found in C code. Plus, my experience with such software is that the sort of higher level vulnerabilities that you’d still see in Python code aren’t ones that the C developer has necessarily dealt with.

How could we check?

Do we agree that this somewhat orthogonal to what I'm writing, though?

Regular sofware development is a lot like that as well. But you can kind of get around that by having Python as the "prototyping language", and anything that's proven to be useful gets converted to a language that's more useful for production.

But I fear that the same folks that decried the use of excel by "the masses" are now just as horrified by the widespread usage of Python! :-)

What's an example that makes it bad? Is it a case of the wrong tool for the job?

For example I understand that garbage collection languages shouldn't be used with real time systems like flight controllers.

Python is designed to be highly dynamic and to allow more code paths to be taken at runtime, through interpreting and reacting to the live data - "late binding" in the lingo, as opposed to the "early binding" of a Rust or Haskell, where you specify as much as you can up front and have the compiler test that specification at build time. Late binding creates an explosion of potential complexity and catastrophic failures because it tends to kick the can down the road - the program fails in one place, but the bug shows up somewhere else because the interpreter is very permissive and assumes what you meant was whatever allows the program to continue running, even if it leads to a crash or bad output later.

Late binding is very useful - we need to assume some of it to have a live, interactive system instead of a punchcard batch process. And writing text and drawing pictures is "late binding" in the sense of the information being parsed by your eyes rather than a machine. But late binding also creates a large surface area where "anything can happen" and you don't know if you're staying in your specification or not.

This is very useful when you're writing unit tests or when you want to monkey-patch a behavior and don't have time for the refactoring that this would deserve.

On the other hand, this means that a module or class, no matter how well tested and documented and annotated with types, could be entirely broken because another piece of code is monkey-patching that class, possibly from another library.

Is it the case? Probably not. But how can you be sure?

Another (related) example: PyTorch. Extremely useful library, as we have all witnessed for a few years. But that model you just downloaded (dynamically?) from Hugging Face (or anywhere else) can actually run arbitrary code, possibly monkey-patching your classes (see above).

Is it the case? Probably not. But how can you be sure?

Cue in supply chain attacks.

That's what I mean by auditing for safety and security. With Python, you can get quite quickly to the result you're aiming for, or something close. But it's really, really, really hard to be sure that your code is actually safe and secure.

And while I believe that Python is an excellent tool for many tasks, I am also something of an expert in safety, with some experience in security, and I consider that Python is a risky foundation to develop any safety- or security-critical application or service.

If I write something in C++ that does a task in 1 second and it takes me 2 days to write, and I write the same thing in Python that takes 2 seconds but I can write it in 1 day, the 1 day of extra dev time might just pay for throwing a more high performance server against it and calling it a day. And then I don't even take the fact that a lot of applications are mostly waiting for database queries into consideration, nor maintainability of the code and the fact that high performance servers get cheaper over time.

If you work at some big corp where this would mean thousands of high performance servers that's simply not worth it, but in small/medium sized companies it usually is.

With Python I can write and test on a Mac or Windows and easily deploy on Linux. I can iterate quickly and if I really need "performance" I can throw bigger or more VPSes at the problem with little extra cognitive load.

I do not have anywhere near the same flexibility and low cognitive load with C++. The better performance is nice but for almost everything I do day to day completely unnecessary and not worth the effort. My case isn't all cases, C++ (or whatever compiled language you pick) will be a win for some people but not for me.

Code is "compute heavy" (could equally be memory heavy or IOPs heavy) if it's deployed into many servers or "the cloud" and many instances of it are running serving a lot of requests to a lot of users.

Then the finance people start to notice how much you are paying for those servers and suddenly serving the same number of users with less hardware becomes very significant for the company's bottom line.

The other big one is reducing notable latency for users of your software.

But sometimes, you do end up writing that compute heavy piece of code. At that stage, you have to learn how to write your own native library :)

Speaking of which, I've written some Python modules in Rust using PyO3, its' a very agreeable experience.

This is the first line in most scientific code:

    import numpy

And OCaml (which offers a productivity comparable to Python) is sensibly slower than Rust or C++.

What slows Python down is generally the "everything is an object" attitude of the interpreter. I.e. you call a function, the interpreter has to first create an object of the thing you're calling.

In C++, due to zero-cost abstractions, this usually just boils down to a CALL instruction preceded by a bunch of PUSH instructions in assembly, based on the number of parameters (and call convention). This is of course a lot faster than running through the abstractions of creating some Python object.

Nah, it’s the interpreter itself. Due to it not having JIT compilation there is a very high ceiling it can not even in theory surpass (as opposed to things like pypy, or graal python).

Python is primarily slowed down because of the fact that each attribute and method access results in multiple CALL instructions since it's dictionaries and magic methods all the way down.

How so? Python is dynamically typed after all and even type annotations are merely bolted on – they don't tell you anything about the "actual" type of an object, they merely restrict your view on that object (i.e. what operations you can do on the variable without causing a type error). For instance, if you add additional properties to an object of type A via monkey-patching, you can still pass it around as object of type A.

Also, if you are interested, “shapes” are the primitives of both Javascript and python jit compilers instead of regular types.

Commonly it creates the .pyc files, so it doesn't really re-parse your code as a string every time. But it does check the file's dates to make sure that the .pyc file is up to date.

On debian (and I guess most distributions) the .pyc files get created when you install the package, because generally they go in /usr and that's only writeable by root.

It does include the full parser in the runtime, but I'd expect most code to not be re-parsed entirely at every start.

The import thing is really slow anyway. People writing command lines have to defer imports to avoid huge startup times to load libraries that are perhaps needed just by some functions that might not even be used in that particular run.

That doesn’t really take any significant time though on modern processors.

But yes python is slow.

However I've seen good python code be faster than bad C code.

I was talking specifically of pure python code (except the python's standard library itself, where it really is unavoidable).

I have code running that reads ~20 bytes, checks the internal status on an hashmap and flips a bit.

Would it be faster in C? Of course.

Would it have taken me much longer to write to achieve absolutely no benefit? Yes.

E.g. people who once wrote "robust" code in Rust but were "outcompeted" left and right by coworkers who churn out shiny new things at 10x the speed.

I'm sure Rust was the wrong tool for the job in your case but I find this type of get shit done argument unpersuasive in general. It overestimates the value of short-term delivery and underestimates how quickly an investment in doing things "the right way" pays off.

If you're dealing in areas with short time limits then Python is great, because you can't sell a ticket for a ship that has sailed.

And I've seen "the right way" which, again, depending on the business may result in a well designed product that is not what's actually needed (because people are really bad at defining what they want)

What's brilliant with Python compared to other hacky solutions that it does support test, type hints, version control and other things. It just doesn't force you to work that way. But if you want to write stable, maintainable code, you can do it.

That means you can write your code without types and add them later. Or add tests later once your prototype was been accepted. Or whenever something goes wrong in production, fix it and then write a test against that.

Oh and I totally agree you should certainly try to "do things the right way", if the business allows it.

The only thing that can compete with it for productivity in the science space is R.

> I'm sure Rust was the wrong tool for the job in your case but I find this type of get shit done argument unpersuasive in general.

Unless you're working on a fire-and-forget project with a tiny time horizon get shit done arguments are blatantly short-termist.

Business owners definitely prefer less bugs, less customer complaints, less support burden, less outages, less headaches. Corner cutting doesn't make economic sense for most businesses and good engineering leadership doesn't have much trouble communicating this up the chain. The only environment where I've seen corner cutting make business sense is turd polishing agencies whose business model involves dumping their mistakes on their clients and running away so the next guy can take the blame.

What business owners actually don't like at all is how long is takes traditional software development to actually solve problems - which then don't really fit the business after wasting a few years of ressources... and the dumping and running away is worse in Java and other compiled software. With Python you can at least read the source in production if the team ran away...

Java (and dotnet, the two big "VM" languages) is somewhat of a strange example for that; JVM bytecode is surprisingly stable and reverse engineering is reasonably easy unless the code was purposely obfuscated - a bad sign on any language anyways.

For an early stage start up this is almost the only relevant factor for success.

> underestimates how quickly an investment in doing things "the right way" pays off.

What time horizon should a startup optimize delivery for? Minutes, hours, days, weeks? Say you're a startup dev in a maximalist "get shit done now" mindset so you're skipping types, tests, any forethought or planning so you can get the feature of the week done as fast as possible. This makes you faster for one week but slower the week after, and the week after, and the week after that.

Say a seed stage startup aims for 12 months runway to achieve some key outcomes. That's still a marathon. It still doesn't make sense to sprint the first 200 meters.

Sounds like a classic web-dev perspective, my customers hate when we ship broken tools because it ruins their work, new feature velocity be dammned. We love our borrow checker because initially you run at 0.5x velocity but post-25kSLOC you get to run at 2x velocity, which continues to mystify managers worldwide.

People use Python in financial applications, Data Engineering and AI/ML pipelines, infrastructure software etc and the 10x speed can be real.

With Python, testing, good hygiene and a bit of luck you can write core that is maybe 99% reliable. It is very, very hard to get to (100-eps)% for eps < 0.1% or so. Rust seems better suited to that.

Anything else, especially if there isn't a huge premium on speed, meh - Python is almost always sufficient, and not in the way.

I don't need rust specifically. It's just that its memory and thread management really help me to continue what I do in python: focusing on my core business instead of technical stuff.

The less I code the better I feel :-)

It's just an application of the Pareto principle. That last 20% of work to make perfect software costs a lot of time. Customers (and by extension, management) do not care how pretty your code is, how perfect your test coverage is (unless your manager is a former developer, then they might have more of an opinion), they care most that you ship it. Bugs are a minor irritation compared to sitting around waiting for functionality they need, as long as you're responsive in fixing the bugs that do come up.

Of course there are at least three dimensions to discuss here: internal quality, external quality and product/feature fit. Lower quality internal code eventually leads to slower future development and higher turnover as no one wants to work with the crappy code base. Lower external quality (i.e. bugs) can lead to customers not liking your product. Interestingly the relationship between internal and external quality is not as direct as one might think. Getting features out the door more quickly (at the expense of other things) can help with product fit. Essentially, like most things, this is an ongoing optimization problem and different approaches are appropriate for different problem domains.

I am tired of having to refactor shiny new things churned out at 10x the speed and that keep breaking in production. These days, if given a choice, I prefer writing them in Rust code, spending more time writing and less time refactoring everything as soon as it breaks or needs to scale.

That's when rewriting the hot path in Go or Rust or Java or C or C++, can pay off and make those skills very valuable to the company. Making contributions to databases, operating systems, queueing systems, interpreters, Kubernetes etc. also fall into that category.

But yeah if you are churning out a MVP for a new business, yeah starting with Python or Ruby or Javascript is a better bet.

(Erlang/Elixir is also an interesting point in the design space, as it's very high level and concise, but also scales better than anything else, although not especially efficient for code executing serially. And Julia offers the concision of Python with much higher performance for numerical computing.)

With Python I can get things up and going very quickly with little boilerplate, but I find that I'm often stumbling on edge cases that I have to debug after the fact and that these instances necessarily happen exactly when I'm focused on another task. I also find that packaging for other users is a major headache.

With Rust, the development time is much higher for me, but I appreciate being able to use the type-system to enforce business logic and therefore find that I rarely have to return to debug some issue once I have it going.

It's a tough trade-off for me, because I appreciate the velocity of Python, but Rust likely saves me more time overall.

(Fwiw I use all three, Python professionally.)

Is a coping canard invented by programmers who can't into more powerful programming languages.

By far the slowest language for developing is PHP, it's even worse than plain C in that regard.

For the most part they don't need to know about concurrency, memory efficiency etc, because they're using a library where those issues have been abstracted away.

I think that's what makes python ideal - it's interoperability with other languages and library ecosystem means less technical people can produce good, efficient work without having to take on a whole bunch of the footguns that would come from working directly in a language like c++ or Rust.

> they're using a library where those issues have been abstracted away.

I work in Python, and while libraries like numpy have certainly abstracted away some of those issues, there's still so much performance left of the table because Python is still Python.

Not everything can be pushed into numpy, and you can still be left with lots of loops in python.

[0] https://numba.pydata.org/

To say nothing of adding yet another long dependancy chain to the language (python 3.11 is still not supported even though work started in Aug of last year).

I do wonder if the effort put into making this slow language fast could have been put to better use, such as improving a language with python's ease of use but which was build from the beginning with performance in mind.

Julia, to use the great-great-grand-op's example, seems to also have a reasonably easy C interop (I've never written any Julia, so I'm basing this off skimming the docs, dunno, it might actually be much more of a pain than it looks like here).

https://docs.julialang.org/en/v1/manual/calling-c-and-fortra...

Plus that you now need to write performant (and safe) C code, which (to me) defeats part of the reason to use Python in the first place.

Because if you compare the benefit to the cost of rewrite from py to C and cost of maintaining/updating C code and possible C footguns like manual memory safety, etc - then there is no benefit left

And the parent was talking about numpy code which is better than stock python, who knows how far back normal python would send you.

They're comparing numpy (SIMD plus parallelism) with straightforward C code and getting a 2-5x improvement.

But there's two schools of thoughts I've heard from people regarding how to think about these bottlenecks:

1. IO/network is such a bottleneck so it doesn't matter if the rest is not as fast as possible.

2. IO/network is a bottleneck so you have to work extra hard on everything else to make up for it as much as possible.

I tend to fall in the second camp. If you can't work on the data as it's being loaded and have to wait till it's fully loaded, then you need to make sure you process it as quickly as possibly to make up for the time you spend waiting.

#-of-users * total-time-saved-per-user > time-spent-optimizing

Then it's worth it. You can even multiply by cost of user per time unit and cost of developer per time unit, to see how much money was saved.

Even in cases where its the same person on both sides, it can still work out. There's an xkcd comic about it, even.

In my opinion, this is the part that Go got mostly right. Concurrency is handled by the runtime, and held behind a very thin veil. As a programmer you don't really need to know about it, but it's there when you need to poke at it directly. Exposing channels as a uniform communication mechanism has still enough footguns to be unpleasant, though.

In an ideal world, I should be able to decorate a [python] variable and behind the scenes the runtime would automatically shovel all writes to it through an implicitly created channel. Instead of me as a coder having to think about it. Reads could still go through directly because they are safe.

If I could have Python syntax and stdlib, with Go's net/http and crypto libraries included, and have concurrency handled transparently in Go-style without having to think about it, that would be pretty close to an all-wishes-come-true systems language. Oh, and "go fmt", "go perf" and "go fuzz" as first-class citizens too.

Someone else in this thread brought up the idea of immutable data structures as a default. I wouldn't mind that. Python used to have frozenset (technically it still does but I haven't seen a performance difference for a while), so extending the idea of freeze()/unfreeze() to all data types certainly has appeal.

This whole take assumes bad intention on both sides. Nobody's job is easy in this situation. Leadership's job is to set everyone up for success. If things go off the rails and end up with months of back and forth leading to nobody being happy despite good intentions and honest effort, then the problem lies with leadership.

Could you expand on this in the context of Rust?

Please don't read this as "rust is difficult and bad", I definitely don't think it is! But its a low level language, and working with it means dealing with complexity that for some tasks just might not be relevant.

Also if I (a programmer) want to write really really fast code I'm probably reaching for tools like tensorflow, numpy, or jax. So there's not much incentive for me to switch to a more efficient language when as near as I can tell the best tooling for dealing with SIMD or weird gpu bullshit seems to be being created for python developers. If you want to write fast code do it in c/rust/whatever, if you want to write really fast code do it in python with <some-ML-library>.

For a very specific definition of the word "fast" at least.

Rather unfortunately, my current bugbear is that Pytorch is... slow. On the CPU. One of the most common suggestions for people who want stable diffusion to be faster is, wait for it, "Try getting a recent Intel CPU, you'll see a real uplift in performance".

This despite the system only keeping a single CPU core busy. Of course, that's all you can do in Python most of the time.

(You can also use larger batch sizes. But that only partially papers over the issue, and also it uses more GPU memory.)

Very limited view which ignores way too many areas.

Really, for almost any X, somebody somewhere has to do X with strict performance requirements (or at very large scale, so better perf == savings)

Most of these python libraries are only fast for relatively large and relatively standard operations in the first place. If you have a lot of small/weird computations, they come with a ton of overhead. I've personally had to write my own fast linear algebra libraries since our hot loop was a sort of modified tropical algebra once.

Whereas before you could get smarter programmers using Python, now because of the exponential growth of Python, the median Python programmer is likely someone with little or no software engineering or computer architecture background who is basically just gluing together a lot of libraries.

The libraries are the killer feature for me.

This myth wasn't even true many years ago, it certainly isn't true today. You can build a mobile app, game, distributed systems, OS, GUI, Web frontend, "realtime" systems, etc in Python, but it is a weak choice for most of those things (and many others) let alone the second best option.

The saying means that for any given problem, there is a better choice, but second best is the language you know which has all of the tools to get the job done, so the answer is probably just a bunch of pip installs, imports, and glue code.

It’s kind of like “the best camera is the one you have with you” — it’s a play on the differing definitions of “best” to highlight the value of feasibility over technical perfection.

Anything that doesn't require high performance that is. Is there any 3D game engine for python yet? I guess Godot has gdscript which is 90% python by syntax, but that doesn't quite count I think.

If people write program logic in python it will run at python speeds. Otherwise you're not really writing python, like nobody says some linux native program is bash because it happens to be launched from a bash script.

But that's how every scripting language obtains good-not-just-decent performance. A strong culture of dropping down to C for any halfway-important library is why PHP's so hard to beat in real-world use, speed-wise (whatever its other shortcomings).

But that's exactly the strength of Python: it's an interface language. It's meant to make pretty sophisticated things like CUDA easy for everyone.

Most of the provided reasons behind Python's popularity are true also for other languages - portable, open source, productive, big community. This can be also said about PHP, Ruby, or Perl back in 2000s. Why isn't Perl as popular as Python?

I don't think it's all about readability or productivity, but about tools that were built over the last 30 years that have been used in academia and now with the boom in ML/AI/Data Science, they made Python an obvious choice to use for the new generation of tools and applications.

Imagine that the boom in ML/AI didn't happen - would Python be #1 language right now?

I don't think there is a single reason, but it sure didn't help that the community self-destructed by trying to make an entirely new language after version 5 and still call it Perl. It took a lot of years to resolve that nonsense, and in the meantime many people moved on.

It also does not help that Perl is a creative language, useful but very much open to many different interpretations. Hiring a perl guy and expecting them to read someone else's code is a crapshoot. The upside to Python's strong cultural opinions on coding style makes it easier for one developer to pick up someone else's code.

> Imagine that the boom in ML/AI didn't happen - would Python be #1 language right now?

Probably not. But it wouldn't be perl, either. Javascript most likely. But the core usage of python for scripting was never predicated on ML popularity, so it would still be a pretty commonly used language. and javascript has many annoying warts too, so I think plenty of people would still choose to write django apps instead of node, whether ML existed or not.

One of the few programming language jokes I've liked enough to repeat is "Python in Perl for people who can't bring themselves to write Perl code"

Python was friendlier for beginners than Ruby the first time I took a real stab at learning to code during a CNY holiday in 2008, but it wasn’t about the language itself. Ruby was harder then because many of the popular libraries and many of the tutorials were written by people who considered Windows support as an afterthought. It’s hard to express how frustrating it was to have my vacation days ticking down, hitting issues in one tutorial after another and having people suggest I install linux on a VM (a process where I hit still more snags).

People learning Python and PHP didn’t hit that hurdle. I ended up learning Flash on my Asus laptop a couple of years later and getting my start that way and not coming back to Ruby until six years later when I was a much more experienced dev.

Perl was significantly more popular at one point, but it slowly lost traction while Python gradually gained traction over the years.

Better ecosystem for numeric computing is definitely a big reason for the success of Python, by the question is why Python gained a foothold in that niche in the first place. It think it is because Python is just a lot more accessible to people with different backgrounds. Perl really grew out of shell scripting as a supercharged alternative to Bash and Awk, but retaining many of the quirks for familiarity. Python on the other hand grew out of research in teaching programming to beginners.

This is a strange article. It's got the talking point about Python that we were hearing about 10 years ago - "tired of those pesky curly brackets in Java, try this new language you might not have heard of: Python!". Who reading the GitHub blog has not heard of Python?

Also, that snippet used in the "What is Python commonly used for" section is strange:

  import antigravity
  
  def main():
      antigravity.fly()

  if __name__ == '__main__':
      main()

It then ends with what seems like it might have been the point of the article, an advert for Copilot. It seems the way to get started writing Python is to write a short comment and then spam <TAB> and let the AI auto-complete your project.

(Also, and perhaps less importantly, looking at the author's GitHub profile I can't see a single instance of Python there. Though I'm not doing a deep-dive as that feel overly picky and there's plenty of private contributions that could well be Python.)

In the end it is just an ad for Github products and not worthy of being on HN frontpage.

That was a talking point closer to 20 years ago, at this point.

   1980-2000 = "30 years ago"
   2000-2010 = "10 years ago"
   2010-2023 = "Recently

The GitHub blog became a strange place recently(-ish). It went from a factual blog describing fancy new GitHub features and interesting technical stuff (as it was a decade ago) to mostly a place full of incoherent marketing fluff like this post (with some real technical content interspersed).

https://github.com/python/cpython/blob/main/Lib/antigravity....

IMO, that example is there to show that there's less required boilerplate required in python compared to java when doing the same thing. And, in particular, none of that boilerplate really matters to what you want to do - print hello world - emphasizing the point of python being simple.

  import antigravity

  pydoc3 antigravity

SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello, world!")?

https://stackoverflow.com/questions/6182964/why-is-parenthes...

That's not a sales pitch for python, it's a sales pitch for the concept of a scripting language; it's like saying "you should really buy a Ford, it comes with four wheels".

I wouldn't say it's friendlier than the alternatives, Perl and Shell scripts sure, but not when compared to Javascript, Ruby or Lua.

Now, if you're talking about libraries, support, etc. then sure, Python wins hands down, but that doesn't make it a better language in itself. I'd say Ruby and Lua are a little bit better as languages.

But then again, I don't care much for the language in itself, so Python is enough for most of my use cases.

In my job I've seen some beginners starting with R, and quickly hating it because they don't feel they can do much on their own, but copy-pasting and then modifying from the examples and the tutorials. And it the changes go too far, everything collapses with cryptic errors. When you show them Python as an alternative, pointing that they shouldn't use it over R for statistics and graphics, they like that they can build ideas from the scratch. That beginner is hooked for life.

The story of Ruby is altogether different: they made the fatal mistake of not defining a C foreign function interface in the standard, otherwise I imagine we'd be seeing numerical computation and ML libraries with a Ruby interface today. Still, Ruby lives on in Metasploit, and in Sorbet and Crystal.

- Lua's standard library is so weak that it makes most other batteries-not-included languages look like they have large, robust, and helpful standard libraries.

- It's got a bit of the quirkiness and gotcha-ability of JavaScript but without its being a language that's impossible to avoid due to capture of a mega-popular platform, which is what propelled JavaScript to ubiquity despite its being kinda shit and unpleasant to work with.

- Tooling's not as good as many other languages.

(FWIW sometimes I write Lua regardless, because it's the right tool for the job)

This to me is extremely plausible, and sad.

In fact I'd argue that python enjoying the success it has, despite probably the worst handling of a version bump in any language (2->3), is a testament to its popularity.

Ruby would have a much bigger market share these days if library path dependence was that powerful.

Chiefly, because big money corp invested massively elsewhere.

Also by now it’s easier to find developers who are cheaper and already (only can) use javascript/python?

I think that actual technical merits are dwarfed compared to other forces at stake here.

https://github.com/elixir-nx/nx

I don't see how it could ever overtake Python, but it could establish itself as a viable niche alternative.

This week our Director ordered a total rewrite of two years of work in Python. His rationale: it's what everyone else uses in this space. No reason specific to our use case, just simply to follow the herd. I realise that a large community translates into easy hiring and rich ecosystems, but I despise the mentality as it promotes a monoculture.

    = be usable, and 

    = provide a set of data structures that an educated programmer *expects to find when scripting*.

Can you say the same about your solution?

There was a time when all you had to learn was Java or C++ or (language or tool here). Somehow, this time when we standardize, it will be different.