Why we defined Testing as a key priority
In 2010, our IT systems were more complex than today in terms of solutions mix as managing the activity of various brands, in various countries, within a larger group holding. As a result, we had significant pains to release projects in part due to integration and testing. At that time, we were finishing the harmonization of the international countries’ IT into a single front-end platform, a single multi-instance back-office and reporting solutions.
We needed to perform the migration while delivering business projects at the same time, therefore we had invested in development of Java applications instead of the legacy COBOL ones, to be able to deliver smaller projects with less risks. At that time, our release cycle ranged from quarterly to monthly, due to the monolith architecture and its complex delivery and manual testing effort. We were still slow with repetitive and error-prone tasks we wanted to get rid of, to reach at least a weekly release cycle.
Our limiting factors were clearly on the testing phase and rework involved in case of regression found, therefore we decided to address this topic as a priority.
Our context and motivations for an internal testing solution
We started to look for commercial and open source solutions at that time, checking with our peers on the current and potentially future solutions. Our requirements were to have on a single solution the test repository, test execution framework and reporting, allowing a fast collaboration between business and IT.
After reviewing the possibilities, our conclusion was that the commercial solutions were either highly focused on test repository, without the execution part, either highly focused on execution but requiring technical scripts implementation. We did not want the business or QA to write, or to have separate people to maintain test scripts. For the open source ones, the only reliable component we found was Selenium, as a web-driver to automate browser actions, mostly used at that time via the plugin for manual tests by QA testers.
Our main customer channels being the website and our internal CRM, we decided at that time to build a POC for an internal testing solution, using Java and MySQL, that would encapsulate Selenium, in order to execute the browser actions. The main requirements remain the same: allow business and IT to work in fast feedback loop on the same platform from test requirements, repository, execution, to reporting.
From the Proof of Concept to the first Release Guard
We had to iterate several times on the internal development. Our first test case was able to place an order on the back-office using our internal CRM system, using local test configuration and performing basic verifications. Satisfied with the technical POC, we dedicated a small number of persons to implement other tests to have a better experience of the Selenium capabilities.
Reaching about twenty test cases on the CRM system and some on the front-end, we quickly saw that we would need to improve the manual configuration, being too slow and technical. We decided to focus on designing the test repository structure and its necessary front-end for the users to easily configure them. with a user-friendly interface. We decided to stay on our MySQL database, and built the front-end using JSP at that time, matching more our internal core of competencies of the teams.
Cerberus Testing product became official
It was the time we decided to give a name to our solution. Our vision was a solution able to perform tests on various types of interfaces – web and devices, local application and APIs – and acting as the guard of our production systems. We decided to name the solution “Cerberus”, in reference to the three headed dogs, guard of the Hell.
One year later, we were able to execute a range of thirty tests for each CRM release, and a small suite for our web platform. The solution did improve, we had at least a simple front-end to administrate the tests, a reporting, and able to manage the complexity of country, environment, platform to execute the tests.
We decided to accelerate deployment for the CRM and Website testing to treat our pain points of low test coverage and repetitive tests. After a year of effort, we reached a 100 tests suite for our core functions, customer creation, search, order entry in the various environments and data set available.
The move to Open Source and adoption acceleration
Having a first solution able to reach a first level in our objectives of user-friendly test repository, execution and reporting, we evaluated internally the possibility of passing the solution to Open Source. We finally agreed to push the solution as open source, knowing other peers were still having the same issues, and convinced that the open source, sharing and communities could accelerate innovation and transversal industry problems. We pushed the project to Sourceforge, Github not being that popular at that time, and started sharing with our first network of peers.
It was also the period when we initiated a new 5-year transformation program at La Redoute, due to the cession of the company from the holding. We had major challenges and projects to implement throughout the whole company, a main one being to harmonize our web platform to one, between international and France. The French activity by itself being the double of the whole international one, we had significant imperatives of quality and value delivery. We decided to deploy Cerberus as the end-to-end functional and regression testing for this major project.
At that time, we did accelerate the Cerberus development, under significant pressure to guarantee our web convergence platform delivery. We had to guarantee the data from multiples sources and technologies, so our connectors were enriched, as well as the actions and controls catalogue. We also delivered a new execution system, enabling parallelisation and execution threading, required to reduce the campaign executions time.
The Open Source Community
After the move to Open-Source, our community was mainly internal, and evolved gradually with some peer’s workgroup. The mailing list was our main way of sharing and helping people to integrate the tool, where the setup, deploy and configuration was not as simple as a container.
Thanks to the feedback, we enriched the tool with the necessary changes identified by the community. We added setup scripts for a one-click deploy for a specific type of configuration, Glassfish and MySQL at that time, also with an administration page to be able to proceed with the solution upgrade from the interface, without using scripts. We also enriched the execution capabilities, campaign management, APIs, local application testing, categories and tags for the test repository.
To facilitate the lifecycle, we simplified the experience of setting up the tests and moved the front-end to a Javascript framework, allowing to configure tests using business languages. We also added the one-click bug opening in bug trackers, allowing fast reproducing of issues between business, QA and IT and shortening the feedback loop for faster releases.
Open Source as a key driver for best-practices sharing
The community was a very powerful way to get transversal feedback from other context to prioritize the product roadmap for the best value. We also had contributions from people member of the community, like the Jenkins plugin we did incorporate as a standard one into the tool. People also helped us in the review of the new front-end, developing new functionalities, performing beta-tests of new versions. We had during several months a weekly presential meeting in the North of France, near our headquarter location, and where major other retailers were evaluating or deploying the solution, like Midas, Norauto, LeroyMerlin.
We wanted to share our experience and spread the solution to people outside of our local region. Not being that much active on social networks, we invested more in attending testing and open source conferences. In 2017, we spoke at the JFTL (Journées Française du Test Logiciel), in 2018 at Open Source Lisbon, Testing Portugal and in local Meetups. We would like to continue this journey and increase our online community, this is a topic we are trying to keep on focus on the product roadmap and development.
Conclusions and Future of Cerberus
The open source repository is now managed on the github repo, with more than 5500 commits and 25 contributors, hoping those figures will increase, maintaining the quality of the community we have today. The discussions are hold mainly through feature requests, you are free to reach the mailing list or the Slack to share with us on the solution. We hope to release soon other initiatives as open source, in the meanwhile, have a nice testing journey !
Sources
Figure 2 : https://www.smartsheet.com/release-management-process