in addition to contributing to upstream python, we
* maintained a stable version of python within google, and made sure that everything in the monorepo worked with it. in my time on the team we moved from 2.7 to 3.6, then incrementally to 3.11, each update taking months to over a year because the rule at google is if you check any code in, you are responsible for every single breakage it causes
* maintained tools to keep thousands of third party packages constantly updated from their open source versions, with patch queues for the ones that needed google-specific changes
* had highly customised versions of tools like pylint and black, targeted to google's style guide and overall codebase
* contributed to pybind11, and maintained tools for c++ integration
* developed and maintained build system rules for python, including a large effort to move python rules to pure starlark code rather than having them entangled in the blaze/bazel core engine
* developed and maintained a typechecker (pytype) that would do inference on code without type annotations, and work over very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)
* performed automated refactorings across hundreds of millions of lines of code
and that was just the dev portion of our jobs. we also acted as a help desk of sorts for python users at google, helping troubleshoot tricky issues, and point newcomers in the right direction. plus we worked with a lot of other teams, including the machine learning and AI teams, the colaboratory and IDE teams, teams like protobuf that integrated with and generated python bindings, teams like google cloud who wanted to offer python runtimes to their customers, teams like youtube who had an unusually large system built in python and needed to do extraordinary things to keep it performant and maintainable.
and we did all this for years with fewer than 10 people, most of whom loved the work and the team so much that we just stayed on it for years. also, despite the understaffing, we had managers who were extremely good about maintaining work/life balance and the "marathon, not sprint" approach to work. as i said in another comment, it's the best job i've ever had, and i'll miss it deeply.
> the rule at google is if you check any code in, you are responsible for every single breakage it causes
i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code in.
this was a very real issue for things like the python runtime or widely used code checkers like pytype or pylint, because if e.g. the new version of python, or some improved check in pytype, started raising failures in a code pattern that was technically wrong but which the existing toolchain did not complain about, you could not release the new version until you had fixed all those new breakages.
contrast this with non-monorepo codebases, where the other code would have had to deal with the fact that "oh, our code works under 3.10 but 3.11 broke it, guess we have to pin our own repo to 3.10 until we fix it", but as the python team we could just say "we support 3.11 now, you need to catch up"
> i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code
Google engineers hate this one weird hack: just send a mail saying "we're deleting/deprecating [everything your product depends on] in [14-180] days, good luck!"
I'm guessing these are related. If every team is potentially responsible for breakages in every product, there's no such thing as a "just chugging along" product and there will be a constant demand to delete products that are not popular within Google.
It has nothing to do with popularity, it has everything to do with teams doing what get them promoted and not having to deal with cross org fallout.
At Microsoft, if there were 12 teams depending on service foo, the team who owned foo had to have a deprecation plan that included sufficient time and support for all 12 teams to get off foo, and directors/VPs would meet and come to terms before the deprecation started.
Google is the opposite. Despite an increasing trend towards top-down direction there's no requirement or support for involving stakeholders. There can be internal products (let's say a data storage solution) and the team who owns it can say "we don't care about this anymore, it's being shut down in 6 months" even if dozens of other teams and millions of end users rely on it, and it's up to those teams to scramble, put other projects on hold, and try to migrate as fast as possible to avoid an outage.
It's frankly demotivating. For the last 18 months, a huge percentage of eng time on my team has gone to either compliance or mandatory migration work i.e. stuff that ABSOLUTE BEST CASE customers don't notice.
Adding to the wonderful writeup by my now-ex teammate (thanks!):
Several of us were/are/TBD also involved in both long term strategic leadership and maintenance of the open source CPython project itself. That direct feedback line from a major diverse needs user into the project and ecosystem was valuable for the world.
The reason I stayed on this team for 12+ years is as zem said. It was an ideal impactful alignment of people, abilities, priorities, and work life balance. My prior teams at Google... were often not.
For the first half of our Python teams existence, there were only ~5 of us. Many early years were spent paying down internal tech debt accumulated from prior years of neglecting to have a strong Python strategy and letting too many do their own thing. Python was one of the very first languages used widely at Google. It was the last major backend language to get a language team.
yes, i definitely should have said more about the cpython leadership work! i was responding to a comment that had already mentioned upstream contributions so i didn't think to highlight it more, but it was a huge contribution to both google and to cpython at large.
well, why don't you guys stay together and continue work? start your own thing! google fired you, yes, but you're still alive and able to communicate. maybe you won't get paid as much as you did at google. but prove to them that you have something of value to provide to society.
My guess: the people in your group have very high salaries and they want to dump them. I would not be surprised if they hire people to make a new team at some point in the future.
I run HR at a small software company reports and my HR person has a lot of friends in HR at FAANGs. For 10+ years we've been talking about hiring and salaries at Google (and others). It's just crazy to me how high salaries have gotten. I know someone that switched from Google to FB when they made $750k at Google and got $1M at FB.
I said all this to say: I've talked to a bunch of people over the last 6 months that know someone laid off and: 1) they had a very high salary, and 2) the group they were in was hiring.
I was your opposite number at AWS for many years. Like you say, it was one of the best jobs I've ever had, but over there it was a career dead end. I wasn't laid off, but I had to leave it to have any hope of growing myself.
I feel for ya, zem; if you ever turn up at a PyCon in person, lemme buy you a drink.
I guess it was in some ways a career dead end at google too, I joined as a senior engineer and never bothered applying for promotion to staff, and most of my teammates were similar. On the other hand, I never felt I wasn't growing myself because I learnt a lot, took on some very ambitious projects, and was trusted with a lot of ownership and input into "what do you feel the most effective thing you could be working on right now is?"
If any of you are interested, feel free to directly reach out to the recruiter Connor Anderson (Connor.Anderson@getcruise.com) or feel free to apply directly.
Please keep in mind that the listed salary range on the job description is only the median base salary range. Not the full total compensation range that includes bonus and other factors to it. We can pay much more than what is listed there from a total compensation standpoint.
> very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)
Is there any place where we could see such tools? A lot of the scripts that I write are contained in a single file, and I'm sure that there would be much to learn from seeing such project scripts. Thank you.
i can't, unfortunately, it was youtube internal code. but by the same token a lot of the performance stuff was specific optimisations for that code and would not really generalise.
Any guess how many SWE-yrs are now going to be spent migrating existing Python projects away from Python, now that Python seems to be unsupported at Google going forward?
Can you say more about the automated refactoring? How did it work? How was it triggered? How much human involvement was there? Are any parts of it public?
can't say too much about it because it mostly used a lot of internal tools, but it was a combination of writing per-file utilities to do specific code transforms (think something like libcst), and then a framework to scan the google codebase, find affected files, and apply the transforms to them.
yes, gated by human reviews based on who owned the code, unless they were completely trivial and then were gated by a single global approver. there were always humans in the loop though.
Ah makes sense. At another MAANG, there was a code mod template job with a half dozen subvariants. I recall using a variant that supported ripgrep piping to sed -i for a simple replacement mod. There were jobs properties controlling how many mods per diff and whether it needed human approval or not. I think I did one batched by 1000 running once every 2 weeks to keep noise down.
> maintained tools to keep thousands of third party packages constantly updated from their open source versions.
Hi Zem could you say more about how you maintained those thousands of third party packages? We are facing the same issue and would love to hear your insight!
sure! it used a collection of tools, the main ones were
* copybara [https://github.com/google/copybara] to mirror between github and our internal repo and transform the source tree layout to our internal conventions
there are internal tools to automate the copybara configuration for python packages specifically, and to map between mercurial and google's version control system, but that's the main idea, and a specific review process for adding new packages.
When was pytype invoked? Runtime? CI/CD? Did it output types to be added to files or did it infer and then enable downstream tasks based on those inferences?
it was invoked manually at build time, and then at CI/CD time (there was no runtime type checking). we did output types to pyi files (the python equivalent of C header files), but mainly we used the inferred types to check the code for type correctness - one of the unique selling points of pytype, and why google was developing its own typechecker in the first place, was that the other available python type checkers would check annotated code for consistency, but would not do anything about type errors in unannotated code. if pytype did detect type errors, the code failed CI and had to be fixed before it was checked in.
we did also have a separate tool that could merge the inferred types back into the source code as annotations, but it never got much uptake.
sorry to hear about the situation at google, would love to help out. we connect laid-off engineers to startups. if anyone from the team is interested here is our discord server
https://discord.com/invite/Bnp5zyc8nM
Frankly, it sounds like a dream job for me too, to be responsible for a particular technology and make sure that it works for all your internal (and external) users.
* maintained a stable version of python within google, and made sure that everything in the monorepo worked with it. in my time on the team we moved from 2.7 to 3.6, then incrementally to 3.11, each update taking months to over a year because the rule at google is if you check any code in, you are responsible for every single breakage it causes
* maintained tools to keep thousands of third party packages constantly updated from their open source versions, with patch queues for the ones that needed google-specific changes
* had highly customised versions of tools like pylint and black, targeted to google's style guide and overall codebase
* contributed to pybind11, and maintained tools for c++ integration
* developed and maintained build system rules for python, including a large effort to move python rules to pure starlark code rather than having them entangled in the blaze/bazel core engine
* developed and maintained a typechecker (pytype) that would do inference on code without type annotations, and work over very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)
* performed automated refactorings across hundreds of millions of lines of code
and that was just the dev portion of our jobs. we also acted as a help desk of sorts for python users at google, helping troubleshoot tricky issues, and point newcomers in the right direction. plus we worked with a lot of other teams, including the machine learning and AI teams, the colaboratory and IDE teams, teams like protobuf that integrated with and generated python bindings, teams like google cloud who wanted to offer python runtimes to their customers, teams like youtube who had an unusually large system built in python and needed to do extraordinary things to keep it performant and maintainable.
and we did all this for years with fewer than 10 people, most of whom loved the work and the team so much that we just stayed on it for years. also, despite the understaffing, we had managers who were extremely good about maintaining work/life balance and the "marathon, not sprint" approach to work. as i said in another comment, it's the best job i've ever had, and i'll miss it deeply.