The event was divided in two talks, the first with speakers from La Redoute, David Frazão and Filipa Nogueira, and the second one from Elastic, with Tiago Costa and João Duarte (Figure 1). The participation of the Elastic team was the culmination of our joint work in: implementation of use cases in La Redoute, participation on the training course last December in Lisbon (Certified Engineer I & II), and many other meetings to best take advantage of the Elastic’ features. During this event, people have learned a lot about La Redoute and Elastic.
Let’s check the lessons learned during the Elastic Meetup.
# 1 From COBOL to Elastic: a DevDataOps in Leiria!
Today, laredoute.io accommodates an IT centre, in Leiria, employing over 100 people from Development, Operations and a newly created Data core. Even though La Redoute has 180 years of history, the technological stack is not that old. In fact, it is quite modern! La Redoute still have some vintage components developed in COBOL, but those are part of its history and make transition to new solutions a very appealing use case and, at the same time, quite challenging.
In broad strokes, La Redoute’s technical stack is organized in four main layers: Platform & Infrastructure, Storage & Database, Application & Integration and Presentation & Visualization. In all layers, La Redoute is developing new solutions using the state-of-the-art technologies, and in parallel, still do maintenance and support for the old components, ensuring the customers have their orders processed and dispatched while focusing on modernising the IT systems.
La Redoute has a miscellaneous set of technologies ranging from the old IBM AS400/Mainframe servers to a new Kubernetes infrastructure to be deployed in hybrid cloud platforms such as GCP and Azure (Figure 2). A similar transition is happening in the Database & Storage layer, moving from the old DB2 to PostGreSQL and ElasticSearch clusters, with some Oracle solutions in-between.
In the Application & Integration, the set is even more diversified. IT teams work with jobs developed in COBOL, web services implemented in SOAP/XML or REST/JSON, Spring Boot applications such as web services and batches, Logstash components that treat business data and logs data from different sources, FileBeats components that handle application logs, a few ElasticSearch clusters created to support different objectives, and other legacy technologies.
An e-commerce platform like La Redoute generates a huge amount of data: from business to infrastructure logs. To analyse data in Presentation & Visualisation layer, we use different tools, from PowerBI to Kibana dashboards. The goal is to check how everything is going, e.g., number of orders created and dispatched, applications not working as expected, among other technical and business needs.
The goal of this brief description was to illustrate that the Elastic stack has an important role in La Redoute IT context as it is used at different levels.
# 2 Elastic stack can help you to understand what is happening in your infrastructure
A major usage of the Elastic stack at La Redoute can be seen in the infrastructure’s use cases that allow to gain visibility about the system and its current status.
In Figure 3, two charts are reporting the number of documents and size of the cluster for two domains: Logs data (left) and Business Application data (right).
You may be surprised to see clusters implemented in old versions of Elastic Search as old as 1.4! You may ask “Why?”. It is just a matter of time, a migration will happen very soon, and those clusters will move to more recent versions. Unfortunately, it is not easy because the upgrade needs to manage the possible impacts on the systems using the clusters (without having a negative impact on the business).
Luckily, Elastic’s support provide tools for the migration implementation of the Cluster, very straightforward to execute.
One of the points mentioned by David during his talk, and one of the most important, is that the logs are really analysed by the teams. Example, after the implementation of a dashboard, based on the logs, reporting several levels of errors, we managed to reduce the errors total by 63%. This happened because feature teams were aware of the errors and used both ElasticSearch and Kibana/Grafana tools to check them and plan the proper fixes.
Another tool using information stored in an instance of ElasticSearch is the new dashboard that provides metrics about the time to deploy a component in La Redoute’s environments, production and non-production. As of today, this dashboard focuses on a certain percentage – around 10% – of the components developed by the team. The goal is to extend it to other applications, but the implementation for older components (COBOL) may be difficult as those have specific deployment processes.
Impressive numbers are displayed in both charts, specially, for the Logs’ clusters (left) which accumulate a total of 6500 million of documents, which in turn represents a total of 20,8 TB of disk space.
One of the points mentioned by David during his talk, and one of the most important, is that the logs are really analysed by the teams. Example, after the implementation of a dashboard, based on the logs, reporting several levels of errors, we managed to reduce the errors total by 63%. This happened because feature teams were aware of the errors and used both ElasticSearch and Kibana/Grafana tools to check them and plan the proper fixes.
Another tool using information stored in an instance of ElasticSearch is the new dashboard that provides metrics about the time to deploy a component in La Redoute’s environments, production and non-production. As of today, this dashboard focuses on a certain percentage – around 10% – of the components developed by the team. The goal is to extend it to other applications, but the implementation for older components (COBOL) may be difficult as those have specific deployment processes.
# 3 Elastic stack is good for your business
As mentioned in the previous lesson, Elastic stack can help us understand what is happening within the system because La Redoute has ways to store and to analyse the logs generated by the ongoing operations. Also, briefly mentioned by David in his talk, it is also possible to use the Elastic stack for Applicational purposes.
As for the business case presented in the meetup – “Offers Data Lake – Pricing in the shopping cart” – What does it mean? It means that the Elastic stack is being used at the Shopping Cart level to obtain information about an article, including “vital” info as the price. So, high availability, quick response times and high scalability are Service Level Objectives (SLOs) for this process. Having no access to price is not acceptable.
The information about an article’s price (+ other details) is encapsulated in the concept of an offer in order to have a sellable article on La Redoute’s platform (Figure 4). Suddenly, there are three concepts that resemble the same: article, offer and sellable article. An introduction to the subject is then required.
First let’s compare an article and an offer. An Article can be simply defined by the basic and inherent characteristics of a product: colour, size, among others. An Offer, on the other hand, encloses indispensable information to sell the article: price, availability and services associated (e.g., credit card, set up of a piece of furniture).
So, basically:
Sellable article = article + offer
After that, a sellable article is used as input to our Discounts engine, and then we have the final price for the customer!
Final price = apply_discounts (sellable article, )
As you can imagine, this is an oversimplified view of the process so you can understand its criticality. Back to the lesson’s purpose: how can the Elastic stack help our business?
Today, the back-office that manages all offers information for La Redoute + Marketplace (don’t forget the +600 external vendors!) comprises a set of relational databases: PostgreSQL and Oracle. In practical terms, this means that the data required in the Shopping Cart is spread across different data sources.
Without looking at the Figure 5, can you idealize a solution where the ElasticSearch component could facilitate the access to the offers’ data?
The solution implemented by La Redoute teams is as it follows: the Shopping Cart accesses to an ElasticSearch cluster instead of the n different databases, and the documents stored in the offers index have the desired format, i.e., it contains the info from all databases. Of course La Redoute has an API layer between the ElasticSearch and the Shopping Cart which allows us to implement certain business rules.
But, how do we load the offers index?
La Redoute has implemented a Logstash component that queries all the data sources, and 1x a day generates a FULL index with the most recent view of the offers.
But, if the offer is updated during the day by the business team, do we need to wait for the next day to have it updated on the index?
No, another Logstash component was implemented to load the DELTAS during the day; it is a component executed N times a day, so the offers will be updated very frequently.
The Elastic stack helps the business by ensuring our SLOs:
- Quick response times in the shopping cart: 1 access is required to get an offer instead of N
- High availability: maintenance operations on our databases don’t impact directly the Shopping Cart; it continues to provide valid information to the customer in case of database downtime.
- High scalability: in case of an event such as the Black Friday, it is “easier” to escalate horizontally the ElasticSearch cluster, in opposition to an increase of Database’s resources.
Technically, to sum up, La Redoute uses ElasticSearch to store the offers index, Logstash to load the offers into the index, and Kibana to visualize the data stored. The activation of Security features is a topic currently in progress.
acraske