Ghosts Are Real
Ghost Inspector Automated Website Testing - Save Time, Money, and quite possibly Souls.
[Editor’s Note: This post was originally published on Medium here.]
I test software for a living but for all intents and purposes, I’m a ghost. If I do my job well, you won’t ever know what I did. If I do it perfectly, you won’t know that I ever even existed.
My name is Cedric, and I am a Quality Assurance Engineer.
Refinery29 is a team of 450 people and growing. I am one of 4 QA Engineers who supports a team of 50 onsite and remote Software Engineers.
The heart of our business is the ability of our writers, editors, contributors, video producers and support staff to publish content 24/7 using a content management system that we developed and refer to internally as “DASH”. We publish somewhere between 700 & 800 new stories per week so if this online printing press of ours were to crash, R29 production would come to a complete standstill.
DASH was built over 5 years ago. Over time, new code was added to old code, then newer code added to that, over, and over, and over again. The Old DASH story editor wasn’t bad but it certainly wasn’t optimal for a company that’s more than doubled its size in the past two years alone.
Roughly 5 months ago, a team of 10 developers, 1 UX Designer, and 1 Product Manager embarked on the monumental task of completely redoing the DASH Story Editor from the ground up. They had to do it using our existing live content and databases, they had to implement this along side a new powerful asset management system and it all had to be done without any interruption to everyday editorial productions.
The new DASH story editor and asset management toolset included dozens of brand new editing features like large batch asset management with duplicate image rejection, batch asset metadata duplications, hi-res auto-image sizing for multiple devices, inline image asset placements, third-party video service integrations, and support for conflict-free multi-user story-editing, to name a few. These updates would touch every aspect of our site. The risks were tremendous. I was the QA Engineer assigned to the project.
Not only were we able to get this done, we got it done without a single fire drill level issue presenting itself. Not one writer delayed, nor a single editor blocked.
I make my living trashing the work of people a lot smarter than me. I don’t mean like the vitriol a hack book critic might spew at a gifted author, no, what I do is much much worse. I take the meticulously labored over babies from the very proud hands of anxious developer parents. I hold these babes up to the sunlight and then I promptly begin to tell them the cruel, dark, ugly truth of everything that is wrong with them. How colicky their code is, how gassy, burpy, and poopy their feature appears to be. I detail with subdued glee the sleepless nights they are likely to encounter because their code never stops crying and a new operating system has just been released that absolutely hates crying. Hates it.
Now I know, even for a metaphorical “software-baby”, this is harsh treatment. Try to understand that sometimes, every now and again, one of these “babies” is actually a terrorist pretending. As cute as they may first appear, they are completely strapped with sticks of bad intention. If I let this little guy through my QA check point well, DASH is to crashes, dust to dust. (You read that right.)
I question the intent of every “loving” code update, modification, and feature that approaches me. No exceptions.
Then, after the feature has passed user acceptance testing (UAT), I still have to ensure that it won’t break any of our existing DASH features or websites through regression and integration testing. This means that for every update, every new feature, every pull request created for the new DASH story editor, I had to go into the old one and make sure the new modifications hadn’t broken any of the existing features our editorial staff depended on.
To do this meant I had to manually conduct hundreds of very repetitive test steps like creating “new” fully formed stories where I inserted valid data in all the required data fields, upload multiple images, videos, and third-party embedded assets. I would have to tag, label and credit uploaded photographs, assign content categories, keywords, regional details, duplicate the finished story to different countries, then I’d have to edit an existing story, insert all those same data and asset types, before finally ensuring that all of those updates could be saved, validated and published to a live website.
Those tests were run for every branch that contained even a minor code update before it was ever allowed to be merged into our production environment. All of those tests were completed by a single entity no one using the story editor even knew existed. An apparition.
A complete — fucking — ghost!
Four months ago, we discovered a new automation tool. A tool so bold and accommodating that when I first used it, I cried. Ok, I may be stretching a bit to make my point but dammit, this tool is just that good.
It’s called “Ghost Inspector”.
Ghost Inspector offers a Chrome Extension that allows you to record and translate every interaction you take on a web page into sequential test steps that you can save and run anytime you wish from your Ghost Inspector Dashboard.
This tool effectively allows you to screen capture your web page interactions and then… well that’s it. Every mouse-click, link selection, button tap, image upload, data field entry or form submission gets converted in real-time to test steps ordered in the sequence of the action taken. You can then save, edit or update your test steps at any point.
Any test you create can also be combined with other tests you’ve created within a test “suite”. (Picture a folder full of tests.) Now you can run OR schedule your test suite or individual tests against any remote developer instance or environment you designate.
All the tests within a suite run concurrent by default which really speeds up your testing cycles. I’ve written dozens of tests and packaged them within well-labeled suites so that ANYBODY can run them. This means any developer, project manager or intern can run a full site wide regression test suite against any remote environment they specify.
So, I’ve described having to manually conduct hundreds of very repetitive test steps before any code update is merged into production. All of those steps are now automated.
When a Ghost Inspector test run has completed, a test status alert is sent to my email along with a link to the test results. These results not only show me the test outcomes but if they fail, they show me at which step. On top of this, there’s a Youtube video of the test interactions performed, embedded within each test. I know, right?
Every completed test within a suite has a video recording that you can visually inspect step-by-step up until the point of success or failure. This feature left me speechless because it empowers even non-technical people with critical information they need to communicate about “what’s broken”.
Additionally, Ghost Inspector offers a host of build deployment integration options. We hooked up our tests to both Slack and Github, so in addition to receiving email alerts I also get “Slack” notifications when tests complete and whenever a pull request branch of code is tagged as “ready-for-testing” our Github integration automatically triggers a suite of Ghost lite-regression tests to run against it. Upon test completion, the test result URL, test status details and the target instance its run against are injected into that pull request as a comment where the developer can immediately review it.
This means every new feature goes through a suite of regression tests before I ever get started.
So, where do I go from here? Well, these tests aren’t going to write and maintain themselves. There’s a lot of evaluation and study required for me to fully begin to understand all of the implications it presents and I already accept that there is simply no substitute for good human judgement or intuition. It is also quite impractical to attempt to automate tests that consider any visual modifications, nor can I expect to fairly evaluate a features “form or usability ” by employing a scheduled pre-written anything.
There also are some features that I’d like to see added to Ghost Inspector like “batch suite” editing where changes can be applied to all the tests in a suite at once. I’d also like to see a more complex hierarchical suite/test capability like perhaps nested tests. For example you can create test-step-modules like for a “login function” in one suite and then import that test-module directly into other suites but a nested-test-module structure that can live within a suite might be more intuitive and a bit easier to manage.
In our planning meetings, I will have to pass new features through an “automate or manual test” decision grid to evaluate when automation makes sense and when it doesn’t. It will also be a challenge to formulate a strident set of best practices for how to organize our suites, structure our tests, and how best to implement them.
There remains an entire engineering team I’ve yet to introduce Ghost to - let alone document for training purposes and all of this continues in the background while I’m still learning about its full potential.
All that said, Ghost Inspector has already proven to be an invaluable QA tool. Thousands of tests are run each month, hundreds of manual testing hours are saved each month and I swear to you, hand to god, my soul is no longer trapped between worlds. I now exist as a fully formed physical being. A Quality Assurance Engineer in the flesh.
– Cedric Gore, QA Engineer
Special thanks to:
Justin Klemm of Ghost Inspector: Justin offers amazing support and is an all around good Earth human.
Matt Anderson VP of Engineering at Audeo: for his assistance with Slack, Jenkins and Github integrations. He’s a generous engineer with a heart of gold.
Greg Shakar Engineering Team Lead, Product and Engineering Refinery29: When I need to do anything “tricked-out” on a test, he’s my goto code guru. He’s also a professor at night because, a Superhero’s gonna do what they do, baby!
Brittnee Cann Team Lead, QA Manager, Product and Engineering. My manager Brittnee stumbled upon Ghost Inspector whilst searching for the Ark of the Covenant. She picked it up, dusted it off, and was nearly blinded by its glow. Not sure if staring into it would turn her to stone, she made the hasty executive decision to quickly hand it to me for an assessment the results of which by now, you’ve certainly read.