Python: Setting up automatic testing with GitHub Actions

Currently I'm working on improving the quality of our Python tools and setting up a good architecture so that the next planned features and automation scripts can fit in well. Now an important step is done using GitHub Actions: Automatic testing on every push to the repository and for every pull request. That helps a lot to avoid regressions: New features should of course not break existing functionality. If you're new to GitHub Actions, I can recommend the quickstart and trying out that GitHub Actions demo on your repository.

Here's now how to set up automatic testing for Python:

1. Write Python tests and put them all into one folder

That's the hardest and most time-consuming part. The Python unittest module is great but still writing good, understandable and meaningful tests is not easy. In our repository still only one third of the code is covered with testing, but that can already give a good idea how it works - check out the test folder in the repo.

Individual tests can be run with python3 -m unittest test_lang.py or simply python3 test_lang.py. All tests together can be run (from the repository root) with

python3 -m unittest discover -s pywikitools/test   (the -s switch tells unittest discover in which folder to look for tests)

2. Set up workflow

Create and edit a file .github/workflows/main.yml in your repository. This is how ours looks like:

name: Run tests

on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  python-tests:
    runs-on: ubuntu-latest

    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2
      
      - name: Install Python uno module (LibreOffice UNO Bridge)
        run: sudo apt-get install python3-uno
        
      - name: Install python packages
        run: pip install -r requirements.txt
        
      - name: Run all tests
        run: python -m unittest discover -s pywikitools/test

In our case we first need to install a package which is fairly time-consuming (~30s) and won't be necessary in many cases. The rest should be self-explanatory.

3. Enjoy

After understanding how it works it's actually quite easy. Now these instructions will be run each time you push to your repository or a pull request is opened and you'll directly see whether the tests were successful. Also you get notified via email if tests fail. Two final remarks:

Increase security: In your repository settings -> Actions -> General set the workflow permissions to read. As we're only running tests there is no need for workflows to have write permissions.

Create a status badge: Now you can add a nice badge to your README as I did. That's not only cool but it helps to be transparent and have an important quality indicator directly up front when you or someone else looks at the repository. GitHub has documentation on that and provides a quick way to copy the necessary code in the "Actions tab" - only in markdown though, for reStructuredText see this link.