Visual Regression Testing with BackstopJS

The Pagoda atop the Pantheon office — @getpantheon

Hi, I'm David Needham

Developer Advocate at Pantheon

davidneedham on GitHub and Twitter

[email protected]

In my free time I like:

Biking with my wife and kids
Board games: is.gd/davidsgames
Volunteering in the community

Agenda

Intro and Setup

Visual Regression Testing

Using BackstopJS

Continuous Integration

Standalone Node App

Q&A + Extracurricular

Setup

You need:

Node.js and npm installed globally

Windows users also need Python installed

BackstopJS installed globally
- npm install -g backstopjs
The Node.js example app
A “live” website that is web-accessible
A “dev” version of that same website (local is great)

Visual Regression Testing

What is Visual Regression?

re·gres·sion

/rəˈɡreSH(ə)n/

noun

a return to a former or less developed state.

When have you been plagued by Visual Regression?

Routine security updates and patches
Theming and site building

Killer Use Case #1: Routine Updates

A screenshot of a WordPress website with updates available

Live (before update):

An animated gif of a website before a change

Dev (after update):

An animated gif of a website after a change

Side by side:

Side by side (mobile):

An animated gif of the website on mobile before and after the change side-by-side

Difference:

Visual Regression: Benefits

"Reference vs Test" Comparison

Reference (production)
Test (environment to compare)

Support for multiple viewports
Easily scan your entire site
Helpful error reporting
Easy to write tests

Limits to Visual Regression

Lots of noise in the testing process
High number of false negatives
Difficulties with dynamic content
Still saves time

Using BackstopJS

BackstopJS

Render with headless Chromey or Puppeteer
Simulate user interactions with simple JS
Browser reports with visual diffs, CLI reports and JUnit reports
Plays nice with CI and source control
Run globally or locally as a standalone packaged app with Node
Incredibly easy to use: just 3 commands go a long, long way!

github.com/garris/BackstopJS

Everything defined in the backstop.json file


			{
			  "id": "backstop_default",
			  "viewports": [
			    {
			      "label": "phone",
			      "width": 320,
			      "height": 480
			    },
			    {
			      "label": "tablet",
			      "width": 1024,
			      "height": 768
			    }
			  ],
			  "scenarios": [
			    {
			      "label": "Homepage",
			      "url": "https://pantheonweeklydemosite.lndo.site/",
			      "referenceUrl": "https://www.pantheondemo.com/",
			      "readyEvent": "",
			      "readySelector": "",
			      "delay": 0,
			      "hideSelectors": [],
			      "removeSelectors": [],
			      "hoverSelector": "",
			      "clickSelector": "",
			      "postInteractionWait": 0,
			      "selectors": [],
			      "selectorExpansion": true,
			      "expect": 0,
			      "misMatchThreshold" : 0.1,
			      "requireSameDimensions": true
			    }
			  ],
			  "paths": {
			    "bitmaps_reference": "backstop_data/bitmaps_reference",
			    "bitmaps_test": "backstop_data/bitmaps_test",
			    "engine_scripts": "backstop_data/engine_scripts",
			    "html_report": "backstop_data/html_report",
			    "ci_report": "backstop_data/ci_report"
			  },
			  "report": ["browser"],
			  "engine": "puppeteer",
			  "engineOptions": {
			    "args": ["--no-sandbox"]
			  },
			  "asyncCaptureLimit": 5,
			  "asyncCompareLimit": 50,
			  "debug": false,
			  "debugWindow": false
			}

Running BackstopJS globally

Navigate to your local project
Initialize Backstop: backstop init
Edit your backstop.json file:
- label: name of your scenario
- url: local or development URL
- referenceUrl: production URL
Create new reference screenshots: backstop reference
Run tests: backstop test

Killer Use Case #2: Theming & Site Building

> It'll just be a quick fix

Narrator: It wasn't.
— I Am Devloper (@iamdevloper) August 9, 2018

Running BackstopJS (notes)

What happens if we change viewports or scenarios?
Regenerate references with backstop reference
Rerun the test with backstop test
Gotcha: Is your updated site different than production?

Continuous Integration

Robots serving platters of food and drink with the quote, What parts of your job can robots do better and faster than you?

Example #1: On Cron

Pantheon Example Terminus Auto Update Script

Check for updates on cron
Generate a new environment
Apply updates and test
Make backups everywhere
Apply or notify you

Get the script

A screenshot of the github repo for Pantheon Example Terminus Auto Update Script

Example #2: On Commit

Automated Workflows in Drupal 8 with GitHub, Composer and CircleCI

Add a module with Composer
Push the branch & make a PR
Build the site and run tests
Notify you on Slack along the way

Get the script

Standalone App

Clone the example repo

We've created an example repo that we're going to clone to give you a jumpstart.

There are detailed instructions in the README to run through this on your own next time.

Today we're going to get started by cloning the repo:

git clone https://github.com/davidneedham/visual-regression-testing-workshop

Running the example node app

Change into the folder
cd visual-regression-testing-workshop
Checkout a new branch
git checkout -b my-hacking-1
Install Node dependencies
npm install
Run the example script
npm run start
Wait for the results (it should open in your browser automatically)

Customizing the example

Edit inc/sitesToTest.js is the list of sites to test

Try adding a section for your own site.
nonProductionBaseUrl is your local / development URL
productionBaseUrl is your production site URL
pathsToTest is the array of URIs to test for each site

Edit inc/backstopConfig.js (the viewports, delay, etc)
Re-run the demo with npm run start

WebOps

WebOps is a set of practices that facilitates collaboration and automates processes to improve the productivity of the whole web team from developers and designers to content editors, stakeholders, and more.

The result is cross-functional web teams empowered to develop, test, and release website changes faster and more reliably.

Q&A

Let's build something!