CN was in need of a feature-rich Article Management System, a Newsletter Management System and an advanced Product Directory. More importantly, they needed a single platform that can house all of their product-vertical Websites like DentalCompare and OpthalmologyWeb, which would in turn make it easier to introduce new features and maintain these Websites. We had developed an all-new vertical management platform for them to address these needs back in 2010, leveraging best of breed open source frameworks like JQuery, RabbitMQ and Memcached, and had successfully migrated several of CN’s smaller online marketplaces to this platform. DentalCompare was our first biggie, in terms of data volume, feature-extensions and scalability demands.

Whilst Calcey had extensive experience in dynamic website development, transferring huge amounts of data proved to be a relatively new challenge for us. There were many thousand records residing on different databases that needed to be transferred to the new site.  Our task was to obtain the data from multiple data sources, cleanse and load the data into the new unified data store. Ensuring data integrity, making sense of the old data formats and transforming these into a format that is meaningful to the new platform were some of the technical challenges that we faced. After looking into many options, we decided to overcome these challenges with SQL Server Integration Services, which made the job less time consuming and more efficient. This project became an opportunity to learn how to migrate a massive bulk of data from a legacy system into a new platform within a tight deadline.

“We started on a solid architectural foundation based on the Microsoft technology stack, and adopted a “living architecture” that got refactored down the road when needed” says Chamindu, our team’s Software Architect. Dhanushka, our Lead Engineer driving the technical solution, says: “we employed all the good practices we had learnt over the years, ranging from Daily Code Check-in through Continuous Integration to Unit Test Driven Development. We setup an automated Selenium Test suite to minimize manual regression testing. We maintained strict programmer discipline throughout the development cycles, employing pair programming and daily code reviews. There were a few tough days when we had to work extra hours, but the awesome end result was well worth the trouble!”

As the appointed Project Manager with a team of 16 engineers, personally I found this project to be an exciting challenge dealing with many unknowns at the beginning. Since this was the first of many large-scale Website migration projects to come, I found that capturing all the potential future requirements from the stakeholders upfront was particularly difficult. So we decided to follow Scrum methodology and managed the workload in fortnightly Sprints. Each Sprint was focused on achieving the immediate known requirements at hand and rigorously tested to ensure quality. Daily stand-up meetings with the team and biweekly Sprint Planning with CompareNetworks ensured that we were on the right course. We managed our tasks on JIRA, a popular enterprise project backlog. The Agile development philosophy proved to be a gem in delivering this baby.

We uploaded the completed DentalCompare Website to a staging environment one week prior to the Live release date, for final feedback by CN stakeholders. We also configured an offline production release to ensure there will be no delay in going Live. On the big day, we ran a “War Room” for 48 hours to continuously monitor the new Website’s performance and to handle any potential issues. We took shifts staying up overnight on standby for defect fixes. In the end, there were only a few minor configuration parameters to be changed during the War Room, and the Website was rocking on Live!

The project was a fun and rewarding experience for all of us at Calcey, and it helped CompareNetworks to take their business to new heights. Besides, CN and Calcey jointly rewarded all of us who contributed to this project with a 3-day vacation at Habarana Lodge, all expenses paid. It was one awesome project.  Best of all, we reused the methodology that we perfected in this project to migrate yet another of CN’s verticals, OpthalmologyWeb, successfully into the new vertical management platform in late 2011.