Mục lục

My Tools and Practices for Quality Python Code

When writing code, the first reflex is to make it work. However, working in a team of professional developers presents a plethora of challenges.

In three successive articles, we will try to address three of these challenges:
Part 1 — Tools and best practices for coding well in Python.
Part 2 — Continuous Integration in Python.
Part 3 — Tools for monitoring, log management, and calculating metrics.

In this article, we will therefore introduce good practices and rules to follow to write quality Python code. We will also, with simple examples, illustrate the use of tools to monitor compliance with these rules.

Code quality?

Several definitions exist, but most agree on the following:

It does what it’s supposed to do: if it doesn’t do this basic requirement, there’s no point talking about code quality.
It does not contain defects or bugs: like any product, before being industrialized, all undesirable defects must be erased. In other words, to speak of high quality, your application must foresee all the scenarios (supposed to treat), and be able to address them.
It’s easy to read, maintain, and extend: No one wants to be in the position of having to read, maintain, or extend shoddy code. That means more headaches and more work for everyone.

How to improve Python code quality?

A good style of development with PEP

For those unfamiliar with Python Enhancement Proposals (PEPs), these are community-written proposals designed to improve certain aspects of Python, ranging from performance to new features, going through the documentation.

Good syntax with PEP 8

The eighth proposal (PEP 8) provides coding conventions for Python code. It is quite common for Python code to follow this style guide. It’s a good place to start because it’s already well-defined. Here are some of the topics that I consider the most important to know about PEP8:

Constants: Are written in uppercase. Ex: My_CONST = ‘…’.
Functions / Variables: In lowercase with possibly underscores. Ex: my_function(), my_variable
Function/method arguments: lowercase and underscore if needed (snake case). Eg: unit_price
Classes / Exception: Must use CapWords( upper camel case ). Recommended not to use the word Class in the name. Eg: MyClass, MyException.
Packages /Modules: Must use lowercase only. Underscores can be used if necessary, but are discouraged. Ex: package or module.py
Indentation: 4 spaces to properly separate the blocks and have a readable code (no tabulation). To do this, you must adjust the code editor, to ensure that when you press the tabulation key, it adds 4 spaces and not a tabulation character).
Line length: No more than 79 characters per line
Empty lines: useful for separating the different parts of the code. It is recommended :
Leave 2 empty lines before the definition of a class.
Leave only 1 empty line before the definition of a method in a class
You can also leave a blank line in the body of a function to separate the logical sections of the function, but this should be used sparingly.
Comments: Start with the symbol #. The complete sentence in English preferably. Avoid comments on the same line of code. Ex :

#My comment
X = x + 1

For more information, you can consult: https://www.python.org/dev/peps/pep-0008/

PEP 257 and docstrings

A proposal describing the conventions for Python’s docstrings, which are strings intended to document modules, classes, functions, and methods. Here is a brief summary.
Surround docstrings with triple-double quotes:

# A comment in a single line
"""A clear description of function in one line."""
# A comment in multiple lines
"""first line
second line
last line
"""

Note that docstrings are intended for users of modules, functions, methods, and classes that we develop. The essential elements for functions and methods are:

what does the function, method, module, …
what he takes as an argument,
what it returns.

The PEP 257 does not require a particular style. NumPy Style Python Docstring, widely used in data analysis, has a style that I find very interesting.
Example :

def function_with_types_in_docstring(param1, param2):
    """Example function with types documented in the docstring.
 
    Parameters
    ----------
    param1 : int
        The first parameter.
    param2 : str
        The second parameter.
 
    Returns
    -------
    bool
        True if successful, False otherwise.
 
    .. _PEP 484:
        https://www.python.org/dev/peps/pep-0484/
 
    """

If the docstrings are consistent, there are tools that can generate documentation directly from code.
For more information, you can consult: https://www.python.org/dev/peps/pep-0257/

All of these guides (PEP8, PEP257) define a way to style code. But how do we apply it? And what about defects and problems in the code, how can we detect them? This is where code quality check tools (Linters) come in.

Code quality control tools (Linter)

To assess the quality of a Python code, that is to say its compliance with the recommendations of PEP 8 and PEP 257, one can use dedicated automated tools such as pycodestyle, pydocstyle, and pylint. To do this you can use the command:

pip install pycodestyle pydocstyle pylint

Also, most IDEs already have built-in liners. Here are some examples:

In addition, there are combo-liters (a composition of several linters), such as Flake8 or Pylama. The latter incorporates a large number of linters including the three mentioned above.

Finally, there are also cool tools to better format the code like Black ( https://github.com/python/black ) or isort ( https://github.com/timothycrosley/isort ). If you are using PyCharm or Atom, dedicated plugins are available.

def make_batches(items, batch_size):
    Current_Batch = []
    for item in items:
        Current_Batch.append(item)
        if len(Current_Batch) == batch_size:
            yield current_batch
             
            current_batch = []
     
    yield current_batch
 
print(list(make_batches([1, 2, 3, 4, 5], batch_size=2)))

After correcting the various syntactic anomalies, we obtain a code conforming to the PEP:

"""This is a test module."""
 
def make_batches(items, batch_size):
   """Generate a list of mini batches.
 
    Parameters
    ----------
    items : list
            list of lists. Each item is an int.
    batch_size : int
            size of each generated mini batch.
    Return
    ------
    list:
        list of mini batch having batch_size
         list(make_batches([1, 2, 3, 4, 5], batch_size=2))
        [[1, 2], [3, 4], [5]]
        """
     
    current_batch = []
    for item in items:
        current_batch.append(item)
        if len(current_batch) == batch_size:
            yield current_batch
 
            current_batch = []
 
    yield current_batch
 
 print(list(make_batches([1, 2, 3, 4, 5], batch_size=2)))

Generation of documentation with Pdoc

First, install pydoc:

pip install pdoc

Then run the command:

python -m pydoc batch_example doc.txt

The contents of doc.txt will be as follows:

NAME
batch_example - This is a test module.

FUNCTIONS
make_batches(items, batch_size)
Generate a list of mini batches.

Parameters
----------
items : list
list of lists. Each item is an int.
batch_size : int
size of each generated mini batch.
Return
------
list:
list of mini batch having batch_size

Notes :

For more sophisticated documentation (md file, html, etc.), you can use another Sphinx module ( http://www.sphinx-doc.org/en/master/index.html ).
A good tutorial for getting started with Sphinx: https://medium.com/@richdayandnight/a-simple-tutorial-on-how-to-document-your-python-project-using-sphinx-and-rinohtype-177c22a15b5b
To go further in the python documentation: https://realpython.com/documenting-python-code/

Behavior-driven development (BDD) and test-driven development (TDD)

In a synthetic way, the BDD is used to formulate the behaviors (functionalities) that must be coded. And the TDD is used to write the solutions to these behaviors.

Indeed, behavior-driven development ( BDD ) is used to answer the question: “What is the desired objective?” “. To do this, we use common words in real life such as: Given, When, Then, and And.
This formulation will describe the behavior of the functionalities that we are looking for.

Example: Stack Specification

Given a stack,
when a new stack is created
then it is empty
and when an item is added to the stack, that item is at the top of the stack.

Test Driven Development ( TDD ) is an approach that puts testing at the heart of our work as developers. There are three rules of TDD to follow, according to Robert Martin (a leader in the world of TDD):

You should write a test that fails before you write your own code.
One should not write a more complicated test than necessary.
One should not write more code than necessary, just enough to pass the failing test.

These 3 rules work together and are part of a process called the red-green-refactor cycle.

Can I use the TDD all the time?

We think the answer is no. The exploratory phase of data science projects is an example of where TDD doesn’t really make sense. After this step, having BDD and TDD in place can help troubleshoot issues more efficiently. Having a clear idea of what is expected of you in terms of functionality and the structure of the code you want is a process that can save you a lot of time.

Code complexity metrics and analysis

This is a very broad subject that we will not go into in-depth here. It will be one of the elements that will be addressed in part 3.
Briefly, the analysis of the complexity and the calculation of the metrics of the code allows the developers to find the problem areas in this code. They need to be refactored. Additionally, certain metrics, such as technical debt, help developers communicate to a non-technical audience why problems occur in a system.

Radon is a tool for obtaining metrics such as line count, cyclomatic complexity, Halstead metrics, and maintainability metrics.

For our previous example, the cyclomatic complexity test displays a very good score for each of the functions (ie, class A).

(reco) C:\recommendationtool>radon cc matrixfactorization.py
matrixfactorization.py
M 18:4 MatrixFactorization.__init__ - A
C 10:0 MatrixFactorization - A
M 62:4 MatrixFactorization.train - A
M 82:4 MatrixFactorization.mse - A
M 96:4 MatrixFactorization.sgd - A
M 123:4 MatrixFactorization.get_rating - A
M 135:4 MatrixFactorization.full_matrix - A

When can I check the quality of my code?

Generally, code quality is checked in one of three situations:

When writing the code

You can use linters as you write code, but this requires setting up your environment.

Before doing a Code Check In

If we are using Git, we can configure GIT hooks to run our linters before making a commit. We can thus block any new code that does not meet quality standards.

Although it may seem drastic, requiring every part of the code to have a linter scan is an important step to ensure continued quality. Automating this filtering at the main entry of your code can be the best way to avoid code that does not respect the recommendations of PEP 8 and PEP 257.

When running tests

It is also possible to place Linters directly in the system used for Continuous Integration (eg Jenkins, Travis-CI, Tox, gitLab-CI, etc.). Linters can be configured to fail to build if the code does not meet the quality standards required of the Linter.

This may seem like a drastic step, especially if there are already a lot of linter errors in the existing code. To combat this, some IC systems allow the build to fail if the new code increases the number of linter errors already present. This way, we can start improving the quality without having to completely rewrite our existing code.

Conclusion

In this article we learned how to write readable Python code by applying PEP best practices and guidelines (8 and PEP 257). We also briefly saw how to use linters, techniques such as BDD, TDD, and complexity analysis.

Although these instructions may seem confusing, following them greatly improves the quality of the code and facilitates teamwork.

To have a python code that lives well in its ecosystem, we will address in the next two parts of this series Python and its best practices the following topics:

Part 2: Continuous Integration with Python: Where we’ll talk about GitLab and GitLab-CI.
Part 3: Monitoring, Log, and metric calculation with Python (Radon, Elastic APM, Promethus, Sentry, DataDog, …Etc).