Testing in district core

Tests are critical to the application, the following is the different test types.

Structure

Test database

We parallize tests which means tests need their own DB via root@%. This should just work but if not try lando ssh -s mariadb-tests, then mysql then CREATE USER 'root'@'%' IDENTIFIED BY 'Lag00n'; GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION; to confirm it worked, run use mysql; then select user, host from user; and you should see something like this (with root having host of %):

+-------------+--------------+
| User        | Host         |
+-------------+--------------+
| lagoon      | %            |
| root        | %            |
+-------------+--------------+

View test coverage

Run lando test-coverage, you can then view coverage in the terminal or visit https://project.lndo.site/coverage for a UI

Test API Module

See more about tests and adding tests to your modules in the TestsApi Readme.

Build Pipeline Failed Steps

If a pipeline build fails, don't trust the 're-run failed steps' option. Tests are run in parallel and this can lead to false-positives. Re-run the full suite before trusting a successful result.

Check changed code locally

To check the files that have changed for code-styling prior to builds

To check your code for code-styling prior to commit:

lando phpcs \`git status --porcelain | grep php | awk 'match($1, /[M]|[??]/){print $2}'`
lando phpcs-fix \`git status --porcelain | grep php | awk 'match($1, /[M]|[??]/){print $2}'`

You can add this to your pre-commit hook as well.

To check your whole branch:

# main branch:
lando phpcs `git diff ..main --name-only | grep php`
lando phpcs-fix `git diff ..main --name-only | grep php`

# or calculate the parent branch
PARENT_BRANCH=`git show-branch | grep '*' | grep -v "$(git rev-parse --abbrev-ref HEAD)" | head -n1 | sed 's/.*\[\(.*\)\].*/\1/' | sed 's/[\^~].*//'`
lando phpcs `git diff ..$PARENT_BRANCH --name-only | grep php`
lando phpcs-fix `git diff ..$PARENT_BRANCH --name-only | grep php`

Cypress e2e testing

We use Cypress for browser and end to end tests. It is recommended you have a browse of their docs to get your head around it. Specifically look at learn.cypress.io for some tutorials.

Installing cypress desktop

• First run lando npm install to install all dependencies and make sure the site is up and running locally
• Run npm run cypress:install NOTE this is NOT inside lando/docker and it means you must have nodejs/npm installed on your host machine

Running cypress desktop

Once installed, just run npm run cypress:open and it will open the cypress desktop app on your PC.

Emulating cypress in CI

Sometimes you might have issues with cypress that only appear in CI. You should first check that Cypress desktop you have installed is the same version as defined in the Core module Resources/shell-scripts/tests/cypress.sh (You can get a specific version via a URL like this https://download.cypress.io/desktop/6.4.0). If so, you might want to try running cypress headless (as it does in CI)

• Run all specs (takes a veeeerrryyy long time) - vendor/bin/district-shell tests/cypress.sh
• Run a single spec - vendor/bin/district-shell Full/Path/To/my.spec.js

Cypress on remote host

If your docker host is remote, ie you ssh into it, then you will need to use X11 forwarding to pass the cypress UI over SSH. On Windows this can be achieved by using Putty or MobaXterm then install and run xming. After this is done, just ssh to your docker host with Putty/MobaXterm and npm run cypress:open.

If using Windows terminal, you need to add export DISPLAY=::11.0 to your bashrc or zshrc on the remote host and ensure your ssh session adds -X as a parameter.

Cypress on Windows/WSL

Ensure you have a vGPU driver installed and the latest wsl (wsl --update in powershell). You then need to install some additional components - assuming Ubuntu:

sudo apt install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev \  
        libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb \    
        x11-apps build-essential ca-certificates libcurl3-gnutls \   
        libcurl4 libcurl4-openssl-dev

Cypress visual regression testing

We use a plugin cypress-image-snapshot for visual regression testing. It's pretty basic to use, at its most basic: cy.get('.hero').matchImageSnapshot(). The first time this runs it will snapshot images to /test/cypress/snapshots, then on subsequent runs it will compare the snapshots and fail the test if the images are different.

Notes with visual regression

• Set your screen size first, you can do this with context() and cy.viewport(), eg

describe('test hero render', () => {
    context('macbook-13 resolution', () => {
        beforeEach(() => {
            cy.viewport('macbook-13')
        })

        it('test 1', () => { ... })
        it('test 2', () => { ... })
    })

    context('iphone-x resolution', () => { ... })
})

• Electron and chrome might take different screenshots so after you have got it all working locally, remove the generated snapshots and then regenerate them via CLI. Eg

./district-shell tests/cypress.sh "Modules/ThemeApi/Tests/Integration/frontend-header.spec.js"

• If you do the above it might change permissions on snapshots dir so followup with sudo chmod -R 777 tests/cypress/snapshots
• Be specific with snapshots, don't snap the whole page, snapshot images will make the repo large so use sparingly

Cypress accessibility testing

For accessibility testing, you can use cypress-axe which can check things like contrast, alt tags, etc.

Basic usage:

cy.visit('/dash')

cy.injectAxe()

cy.checkA11y(null, {
    includedImpacts: ['critical']
}, (violations) => {
    cy.log(violations)
    expect(violations.length).to.eq(0)
})

Custom migrations in php tests

There is a helper trait available RefreshDatabaseWithCustomMigrations which allows you to run custom migrations, just use that trait, then define a $customMigrations array, keyed by the module name and the value is an array of migration paths.

eg

use RefreshDatabaseWithCustomMigrations;

protected array $customMigrations = [
    'InteractionsApi' => ['Tests/Database/Migrations'],
];

Issues and fixes when testing

Wiping the test database

Sometimes cypress (or even php tests) might give errors when it has a dirty test db. Eg Base table or view not found. The following command will fully nuke the tests db volumes, recreate and seed which normally solves db issues in tests

./vendor/bin/district-shell fixes/fix-broken-testdb.sh

Flakey PHP Tests in CI

CI runs tests in parallel via Pest which can sometimes cause flakey tests due to refreshDatabase not actually wiping the tables. This is done for performance reasons as running all migrations when not required is expensive. However, this can lead to unexpected results if you have a test that relies on a fresh DB.

To get around this, there are a few things to try:

1. Ensure your Models table is empty in the setUp method of your test. You can do this with the deleteAll method. Eg Content::deleteAll();
2. For non model tables, eg settings, ensure your test sets the DB value before testing and if you want to ensure the defaults are correct, isolate that into its own test where the migration/seeding is run.
3. Lastly, if you just want to force refreshDatabase to nuke everything each test (making it slower) then you can do something like this

    public function setUp(): void
    {
        $this->forceRefreshDatabase();
        parent::setUp();
    }