In this blog: Learn about the 10 Python tools that are a must-have for every developer! Download and install the “Top 10 Tools to an Easier Life” build, which contains a version of Python and all the tools listed in this post so you can test them out for yourself.
As a developer, it sometimes feels like my list of sprint tasks is infinite. Part of the problem is that many of the tasks are ones that keep coming up over and over. That’s where Python Tools come in. Python is a great all-purpose language that reads easily (almost like English) while still being incredibly powerful. When it comes to tackling those recurring tasks, Python tools, libraries and supported IDEs are so numerous you’re sure to find a package to help fix your problem. And if you can’t find it yourself, Python has fantastic community support.
This post introduces a few Python tools that I always keep around to make my life easier by helping to solve key tasks I need to get done on a regular basis. Let’s dive in.
#1 Simple Web Frameworks with Flask
Need to set up a web server? Do you have two seconds? Because that’s how long it takes to start a simple web server with Python:
python -m http.server 8000
That’s it, and you have a working server that you can connect to from your browser. But for a basic web application, this might be too simple. Enter Flask.
Flask is a micro web framework built with Python. It’s ‘micro’ because it does not have any database abstraction layer, form validations, or mail support. Fortunately, it has a large set of extensions that you can plug and play with, and it’s perfect if all you want to do is serve up a simple API.
To create an API server with Flask, use the following script:
from flask import Flask
from flask import jsonify
app = Flask(__name__)
@app.route('/')
def root():
return jsonify(
app_name="Top 10 Python Tools",
app_user="ActiveState"
)
To execute the server:
FLASK_APP=flask.py flask run
And finally, when you visit the URL in a browser you should see the following JSON:
{"app_name" : "Top 10 Python Tools", "app_user" : "ActiveState"}
#2 Web Scraping with Scrapy
Scrapy is a power tool that lets you “scrape” or more precisely, extract information from websites. When large amounts of information need to be pulled out from multiple websites or web pages, it’s not efficient to do it manually. Instead, developers automate the process with “spiders” or “crawlers” that scrape web pages for information.
Scrapy provides easy-to-use methods and packages to extract information using HTML tags or CSS classes. You can start the Scrapy Shell, which is an interactive mode that lets you play around with different methods, by issuing the following command:
scrapy shell
As a simple exercise, let’s try to extract the value of the search button on the Google homepage. To do this, we need to find the class that the button uses. A simple “Inspect Element” reveals that the class is “gb1”.
In the interactive Scrapy Shell, do the following:
response = fetch("https://google.com")
response.css(".gb1::text").extract_first()
==> "Search"
It’s as simple as that!
#3 API Calls with Requests
Requests is a powerful HTTP library. With it, you can automate just about anything to with HTTP requests, including the automation of API calls so you no longer have to make them manually using a rest client. It comes with useful features such as authorization handling, JSON/XML parsing, and session handling.
Let’s look at a quick example of accessing Github’s API, which is behind an authorization wall:
import requests
requests.get('https://api.github.com/user')
==> <Response [401]>
We got a 401 “Unauthorized Error” message because we tried to access the API without passing the required authorization credentials. Let’s try it again, but this time we’ll submit a valid username and password.
import requests
requests.get('https://api.github.com/user', auth=('user', 'pass'))
==> <Response [200]>
This time we get a 200 “Ok/Success” message. The functionality offered by the Requests package is both broad and deep. For more information on its capabilities, refer to the documentation.
#4 Command Line Packaging with Click
Click is a Python package that can be used to create beautiful command line interfaces.
As developers we write many scripts that make our lives easier, such as fetching external IP addresses, or making a server ping to check if it’s alive, or to simply look up the time. Of course, to run any script, you have to navigate to the directory and then execute it. Want to pass arguments? Forget it! By the time you parse all the user options, you’ll give up.
With Click you can package any Python script that exposes functionality to the command line. Once packaged, you can access your script right from the terminal.
Let’s looks at an example:
import click
@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
help='The person to greet.')
def hello(count, name):
"""Simple program that greets NAME for a total of COUNT times."""
for x in range(count):
click.echo('Hello %s!' % name)
if __name__ == '__main__':
hello()
The method option exposes its argument names to the command. Here, there are two arguments exposed: count and name.
And finally to call our script:
python hello.py --count=3 Your name: John Hello John! Hello John! Hello John!
#5 Automated Testing with Selenium
Selenium is a testing framework to write automated test cases. Although it is written in Java, the Python package provides an API-like access to almost all Selenium functions.
Selenium is typically used to automate the testing of an application’s UI, but you can also use it to automate tasks on your machine like opening a browser, dragging and dropping files, and so much more.
Let’s look at a quick example that shows how to open a browser and visit Google’s homepage:
from selenium import webdriver import time browser = webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe") website_URL ="https://www.google.co.in/" brower.get(website_URL) refreshrate = int(15) # This would keep running until you stop the compiler. while True: time.sleep(refreshrate) browser.refresh()
Now this script refreshes the Google homepage in the browser every 15 seconds.
#6 Data Analytics with Pandas
Pandas is a simple yet powerful data analytics tool. You can use it to read large amounts of data, clean it up, and perform statistical analysis on it. You can quickly generate summaries for the data, or even split it up.
Once you’re done analyzing the data, you can also visualize it with external libraries like Matplotlib.
The awesome thing about Pandas is that it’s built on top of NumPy, which is yet another amazing data analysis tool used to perform numerical methods on data. This means that most NumPy methods are functions that are already available within Pandas.
For more information, including many examples of how to work with Pandas, refer to my previous post on How To Clean Machine Learning Datasets Using Pandas.
#7 Generating Fake Data with Faker
This is hands-down the most useful tool in my kit! Whenever I need to fill up placeholder text or add dummy data to websites, I just use Faker.
With it you can generate fake names, addresses, descriptions, and more! For example, the following script creates a contact entry, complete with name, address, and some description text:
from faker import Faker fake = Faker() fake.name() # 'Lucy Cechtelar' fake.address() # '426 Jordy Lodge # Cartwrightshire, SC 88120-6700' fake.text() # 'Sint velit eveniet. Rerum atque repellat voluptatem quia rerum. Numquam excepturi # beatae sint laudantium consequatur. Magni occaecati itaque sint et sit tempore. Nesciunt
#8 Image Processing with Pillow
A lot of times I need to modify images in some way to make them more suitable for purpose, such as blurring details, combining one or more images, or creating thumbnails. As a developer, my go-to tool is not a GUI application like Photoshop, but rather a powerful Python image processing tool called “Pillow“.
I often combine my homemade Pillow scripts with Click, and then access them directly from the command line. Invaluable for speeding up repetitive image processing tasks.
Let’s look at a quick example to blur images:
from PIL import Image, ImageFilter
try:
original = Image.open("Lenna.png")
# Blur the image
blurred = original.filter(ImageFilter.BLUR)
# Display both images
original.show()
blurred.show()
blurred.save("blurred.png")
except:
print "Unable to load image"
Simple and straightforward.
#9 Date and Time Parsing with Pendulum
It’s never fun to work with date and time formats. I absolutely detest working with timezones and obscure time jobs. Although the built-in Python datetime module does a fairly decent job, Pendulum is a powerhouse! It has a more intuitive interface that allows for quick processing. It supports timezone conversion, datetime manipulations and formatting.
Let’s look at a quick example that allows you to get the time in 3 different time zones, and then manipulate UTC:
from datetime import datetime
import pendulum
utc = pendulum.timezone('UTC')
pst = pendulum.timezone('America/Los_Angeles')
ist = pendulum.timezone('Asia/Calcutta')
print(type(utc))
print('Current Date Time in UTC =', datetime.now(utc))
print('Current Date Time in PST =', datetime.now(pst))
print('Current Date Time in IST =', datetime.now(ist))
print(type(datetime.now(ist)))
===>
<class 'pendulum.tz.timezone.FixedTimezone'>
Current Date Time in UTC = 2020-02-29 09:16:45.031461+00:00
Current Date Time in PST = 2020-02-29 02:16:45.031501-07:00
Current Date Time in IST = 2020-02-29 14:46:45.031555+05:30
<class 'datetime.datetime'>
utc_time.add(years=1)
utc_time.subtract(months=2)
print('Updated UTC Time', utc_time)
==>
Updated UTC Time 2020-04-29T09:16:45.031608+00:00
#10 Code Template Creation with CookieCutter
Cookiecutter is a glorified cheatsheet! It’s a command-line utility that creates projects from “cookiecutters” which are project templates.
With it, you can create project templates and distribute them to your team (or open source them). Now all team members can use your project as a base version for their own, and just make modifications to suit their needs.
If you maintain a Python project, or are interested in publishing a project to PyPI, one of the most used cookiecutters is audreyr/cookiecutter-pypackage. With it, you get a “skeleton” Python package that includes tests, distributions, and documentation, and then just tailor it for your own package to complete your project and even automatically publish it to PyPI.
These Python Tools should be in every developer’s toolbox
The old adage that states, “if you have to do a task more than once, automate it” is still true today. Pythonistas have taken this saying to heart and provided a wealth of packages in the Python Package Index (PyPI) that you can use to help automate all those manual tasks that suck up time, and suck the fun out of being a developer.
As such, Python is a very versatile language and something that should be in every developer’s toolbox.
To get started, you can:
- Create a free ActiveState Platform account and download ActivePython, which is a prebuilt version of Python containing hundreds of packages that can help you solve your common tasks, or
- Download and install the “Top 10 Tools to an Easier Life” build, which contains a version of Python and all the tools listed in this post so you can test them out for yourself:
- Install the State Tool on Windows using Powershell:
IEX(New-Object Net.WebClient).downloadString('https://platform.activestate.com/dl/cli/install.ps1')Or install the State Tool on Linux / Mac:
sh <(curl -q https://platform.activestate.com/dl/cli/install.sh) - Run the following command to download the build and automatically install it into a virtual environment:
state activate Pizza-Team/Top-10-tools-to-an-easier-life
- Install the State Tool on Windows using Powershell:
Related Blogs:
