Background:
Visual CSS regression testing is where you screenshot (part of) one version of a webpage in a browser and compare it against a screenshot of the previous version of the same webpage in the same browser. This prevents unintentional changes from happening due to browser version upgrades or changes in the page itself (e.g. refactoring CSS), because we can diff the two screenshots and flag the new version for review by a human developer if there are any differences. If the difference(s) are intentional, the developer approves the change and marks the new screenshot as the new norm/reference/baseline screenshot. By taking screenshots of the same page in different browsers on different platforms, we can extend this to cross-browser cross-platform CSS regression testing.
With that long explanation now out of the way:
What is the best way to do cross-browser cross-platform CSS regression testing?
I’m trying to see what might be usable for Bootstrap.
2
You can write a small shell script (~5 lines) to launch a browser and capture the window with imagemagick’s import. I use this for Firefox:
firefox -no-remote "$1" &
sleep 10
window_id=$(xwininfo -tree -root | grep Navigator | awk '{print $1}')
import -window $window_id "$1.png"
wmctrl -c Firefox
The only unusual thing here is using wmctrl instead of kill to close Firefox, so it won’t go into recovery mode next time it runs. It should be pretty obvious how you can modify this for Chrome or other browsers (you may need to play around with xwininfo to figure out how to find the browser’s main window).
You can add a line or two to compare the new image with the previous one using imagemagick’s compare.
I’m interpreting this question as asking how to do this rather than asking for off-site resources, since that would be off-topic. There are a number of scripts floating around on the web that do more or less the same thing.
As of December 2014, the best options for cross-browser CSS regression testing seem to be:
- CrossBrowserTesting.com
- Proprietary cloud service; Costs money
- Sounds like it can do the diff-ing and approval management and norm/reference/baseline tracking for you? I haven’t actually tried this service out.
- WebdriverCSS
- A Selenium-based tool that offers a web UI to track, review, and approve differences
- Seems to be backed by Sauce Labs somewhat
- Wraith-Selenium
- A fork of Wraith that adds Selenium support. The original Wraith is limited to CasperJS (i.e. Firefox/Gecko & WebKit only).
- Downside: Not sure if this project is established enough yet to survive long-term.
- Depicted—dpxdt
- Doesn’t support cross-browser testing directly, but it does offer a “Diff My Images” option to import screenshots taken from other tools (e.g. Selenium). (The default setup only uses PhantomJS.)
- Offers a web UI to track, review, and approve differences
For cross-OS tests for the Selenium-based options, you either need to setup your own cluster of test machines or use a for-pay cloud option such as Sauce Labs, Browserstack, etc.
Both Sauce Labs and Browserstack currently offer free accounts for open-source projects.