LuxSE IT Series #2

"We process and store annually over a hundred times the amount of documents we started with 10 years ago," says Kevin Lloyd. "The NoSQL architecture we have put in place gives us the flexibility and scalability we need to meet our objectives and obligations in this field, especially since the database system we have adopted, MongoDB, is the premier database for document storage. This is one of the strong points of our new architecture."

NoSQL databases allow developers to store massive volumes of structured, semi-structured and unstructured data, while offering great flexibility. They also scale easily with large amounts of data and high user loads. NoSQL databases have gained significant traction since the first models were tested 20 years ago, boosted by the sustained decrease in the cost of data storage. In the past years, software development has increasingly reflected the new and evolving needs of developers and users in terms of data storage. NoSQL databases are a testament to this new approach, as they set out to optimise the productivity of developers and ensure a faster delivery to the end-user.

"The new projects that we have based on NoSQL technologies are in two areas," explains Kevin Lloyd. "One is document storage and processing, the other is building and maintaining the database state during that process. As microservices process different phases of a document, MongoDB maintains the process state for each phase. If there is a system failure during a phase or the microservice needs to reprocess data, it reads the MongoDB database and restarts from the last successful phase. This is a key part of our future process management capabilities."

"Within the broader framework of our migration to the cloud, we are moving from a traditional, monolithic SQL architecture to a more decentralised and agile microservices environment," explains Kevin Lloyd.

Cloud computing offers the ability to distribute data across multiple servers to make applications scalable and resilient. Moreover, in an environment of rapid change and constantly increasing data volumes, developers need to be able to make changes throughout their software stack and in the database model itself. This requires considerable flexibility, which is a key feature of NoSQL databases.

"In our journey to build a modern, microservices-oriented architecture, NoSQL is one of the key instruments for data processing," says the CTO of LuxSE. "However,"  he adds, "it's just one additional tool in our toolbox. We are also using traditional SQL for other components of our information system and in other areas of the organisation, for our business intelligence architecture for instance. There will always be a need for us to use traditional SQL, at least in the foreseeable future."

"Change management and training are part of the transformation that we are going through," Kevin Lloyd underlines. "We historically had a lot of traditional SQL expertise. But as we move forward, not only with NoSQL but also with other technologies that we are adopting, there is definitely a learning curve for our staff. Architects and developers must keep pace with rapidly changing and evolving technologies. It's a constant learning process not only to understand the changes in the capabilities but also to get the most from them."

"Of course, some people embrace change at different rates than others," Kevin Lloyd admits. "Some jump on the opportunity and take the lead while others need more time to assimilate, integrate, and eventually take advantage of the new possibilities opening up to them. But at the team level, and even through the entire organisation, you can feel that people are eager to move forward, that they understand the improvements and the new capabilities that come with that architecture. And from a data processing and efficiency perspective, I don't hear anybody complaining that we are leaving traditional SQL behind," he concludes.

Interview by Michael Renotte, originally published as part of LuxSE’s article series on IT One.