Visual Regression Testing with BackstopJS

Brought to you by the folks at Pantheon

Visual Regression Testing with BackstopJS

The Pagoda atop the Pantheon office
@getpantheon

Hi, I'm David Needham

Developer Advocate at Pantheon

davidneedham on GitHub and Twitter

[email protected]

In my free time I like:

David Needham

Agenda

Intro and Setup
Visual Regression Testing
Using BackstopJS
Continuous Integration
Standalone Node App
Q&A + Extracurricular

Setup

You need:

Visual Regression Testing

What is Visual Regression?

re·gres·sion

/rəˈɡreSH(ə)n/

noun

  1. a return to a former or less developed state.

When have you been plagued by Visual Regression?

Killer Use Case #1: Routine Updates

A screenshot of a WordPress website with updates available

Live (before update):

An animated gif of a website before a change

Dev (after update):

An animated gif of a website after a change

Side by side:

An animated gif of the website before and after the change side-by-side

Side by side (mobile):

An animated gif of the website on mobile before and after the change side-by-side

Visual Regression:
The New Pixel Perfect

A screenshot of a generated visual regression report

Difference:

A screenshot of a website showing the differences highlighted

Visual Regression: Benefits

A website on multiple devices

Limits to Visual Regression

Static noise

Using BackstopJS

The BackstopJS logo on a background

BackstopJS

github.com/garris/BackstopJS

Everything defined in the backstop.json file


			{
			  "id": "backstop_default",
			  "viewports": [
			    {
			      "label": "phone",
			      "width": 320,
			      "height": 480
			    },
			    {
			      "label": "tablet",
			      "width": 1024,
			      "height": 768
			    }
			  ],
			  "scenarios": [
			    {
			      "label": "Homepage",
			      "url": "https://pantheonweeklydemosite.lndo.site/",
			      "referenceUrl": "https://www.pantheondemo.com/",
			      "readyEvent": "",
			      "readySelector": "",
			      "delay": 0,
			      "hideSelectors": [],
			      "removeSelectors": [],
			      "hoverSelector": "",
			      "clickSelector": "",
			      "postInteractionWait": 0,
			      "selectors": [],
			      "selectorExpansion": true,
			      "expect": 0,
			      "misMatchThreshold" : 0.1,
			      "requireSameDimensions": true
			    }
			  ],
			  "paths": {
			    "bitmaps_reference": "backstop_data/bitmaps_reference",
			    "bitmaps_test": "backstop_data/bitmaps_test",
			    "engine_scripts": "backstop_data/engine_scripts",
			    "html_report": "backstop_data/html_report",
			    "ci_report": "backstop_data/ci_report"
			  },
			  "report": ["browser"],
			  "engine": "puppeteer",
			  "engineOptions": {
			    "args": ["--no-sandbox"]
			  },
			  "asyncCaptureLimit": 5,
			  "asyncCompareLimit": 50,
			  "debug": false,
			  "debugWindow": false
			}

Running BackstopJS globally

  1. Navigate to your local project
  2. Initialize Backstop: backstop init
  3. Edit your backstop.json file:
    • label: name of your scenario
    • url: local or development URL
    • referenceUrl: production URL
  4. Create new reference screenshots: backstop reference
  5. Run tests: backstop test

Killer Use Case #2: Theming & Site Building

Boy with hands over his eyes

Running BackstopJS (notes)

Continuous Integration

Robots serving platters of food and drink with the quote, What parts of your job can robots do better and faster than you?

Example #1: On Cron

Pantheon Example Terminus Auto Update Script

Get the script

A screenshot of the github repo for Pantheon Example Terminus Auto Update Script

Example #2: On Commit

Automated Workflows in Drupal 8 with GitHub, Composer and CircleCI

Get the script

DrupalCon Seattle 2019

Standalone App

Clone the example repo

We've created an example repo that we're going to clone to give you a jumpstart.

There are detailed instructions in the README to run through this on your own next time.

Today we're going to get started by cloning the repo:

git clone https://github.com/davidneedham/Visual-Regression-Testing-BackstopJS-demo.git
A screenshot of the repo we're cloning

Running the example

  1. Change into the folder
    cd Visual-Regression-Testing-BackstopJS-demo
  2. Checkout a new branch
    git checkout -b my-hacking-1
  3. Install Node dependencies
    npm install
  4. Run the example script
    npm run start
  5. Wait for the results (it should open in your browser automatically)

Hacking the example

  1. Edit includes/backstop.json
  2. Rerun the script with npm run start

Running the multiple site example (pt 1)

  1. Commit your existing changes
    git commit -am "All of my changes"
  2. Checkout the branch
    git checkout multiple-sites
  3. Checkout a new branch to work on
    git checkout -b my-hacking-2
  4. Re-install Node dependies
    npm install
  5. Rerun the example script
    npm run start (and type a site name from the list when prompted)
  6. Wait for the results

Running the multiple site example (pt 2)

  1. Run the command npm run start -- --all (note the extra --)
  2. Wait for the results (from multiple sites!)
  3. Edit includes/sitesToUpdate.js (list of sites to test)
    • Try changing to one (or more) of your sites
    • BackstopReferenceBaseUrl is your local / development URL
    • BackstopTestUrl is your production site URL
    • pathsToTest is the array of URIs to test for each site

Running the multiple site example (notes)

Q&A

Let's build something!