How toTrends

Efficient Engineering: How We Used Talend To Supercharge Business Intelligence.

Despite the availability of a multitude of tools, data can be quite a beast to tame. But, the world that we live in is such that ‘data has become the new oil’, especially when it comes to business. Today, even businesses have evolved to the point where they consider data as their competitive advantage. From Amazon to Google, Spotify, and Tesco, the examples are numerous.

The Problem

However, large volumes of data can make it extremely hard to glean information. This was a recent problem faced by one of Calcey’s very own European clients. The client is in the business of providing cloud-based Point of Sale (POS) solutions to independent restaurants in Northern Europe.

As it set about scaling its operations by signing up new restaurants, the company understood that the sheer volume and complexity of data rendered analysis (in the traditional sense) a wasteful affair. To understand this problem better, consider how a standalone restaurant stores its transaction data. There could be hundreds of SKUs, all recorded using a naming convention chosen by the owner of the restaurant. The data would most likely be stored in a proprietary database, or even on Microsoft Excel. When you consider how a cloud-based solution provider will now have to aggregate all this data across hundreds of restaurants in many different municipalities, the complexity of the task at hand becomes apparent.

The legacy system our client had to contend with before they approached us creaked under the weight of the data it had to bear. Database timeouts were common, and it took around fifteen minutes for a single report to be compiled. The client had to also resign themselves to generating only daily reports since the legacy system could not aggregate data to provide a weekly or monthly report.

So, how does one sanitize and unify all this data, so that actionable information can be gleaned at the click of a button?

Our Solution

In consultation with the client, we opted to conduct a pilot using the data set belonging to a single restaurant. Since unstructured data must first be sanitized, we chose Talend Cloud as the overall data integration and governance platform, primarily because of its flexibility and speed. Talend’s support to integrate third-party business intelligence (BI) tools was also a definite advantage. This allowed Calcey’s engineers to map the database structure to a set of API endpoints, thereby allowing the BI tool to access a dataset asynchronously.

The proposed system architecture

Second, we opted to use HSQL-DB to improve query performance. By using HSQL-DB, our engineers were able to create a memory cache of the dataset, which had the advantage of improving the speed of the API and improving the application’s performance, while reducing the load on the back-end infrastructure. As a result of this structure, Calcey’s solution was able to deliver a much welcome cost saving to the client.

How the caching works
The caching mechanism within Talend

The Results
By virtue of using an in-memory database to crunch the data, we managed to shorten the time it takes for our client to generate a report to mere seconds, compared to the fifteen minutes it took previously. The in-memory database structure also allows for real-time filtering of data. Additionally, we were able to integrate the database with Power BI through the Talend API, which granted our client the ability to generate deep, detailed, and actionable business insights.

How the API works
The API within Talend

Since the API works by obtaining data directly from the cache memory, we undertook to build a job within Talend (i.e. an updater module) which automatically runs according to a predetermined schedule, thus saving time and reducing the workload of the system administrator.


3D authentication is set for mass adoption in EU in 2 months. Are you ready?

This September, Europe will see the introduction of new requirements for authenticating online payments, as part of the second Payment Services Directive (PSD2). These requirements, also known as ‘Strong Customer Authentication’, are going to significantly change how online retailers process payments within Europe. Here at Calcey, we do a lot of work with European clients, who have had to migrate to 3D Secure-compliant processes. Here are a few things which we have learned along the way.

What is Strong Customer Authentication (SCA)?

The European regulators introduced SCA as a method to reduce fraud and make online transactions more secure. Once SCA becomes legally binding from September 2019 onwards, merchants (especially those who conduct transactions online) will have to build an additional authentication component into their checkout flow. For SCA to work properly, every authentication request has to have any two of the following:

  1. Something the customer knows (e.g. PIN number or a password)
  2. Something the customer has (e.g. a hardware token, or a phone)
  3. Something the customer is (e.g. a fingerprint or face recognition)

From September 14 onwards, banks will be able to decline transactions which don’t meet the SCA criteria.

How SCA Works / Credit: WP Simple Pay

How Authentication Works

Currently, the most popular way of authenticating a card payment is via 3D Secure 1— a protocol supported by a vast majority of cards globally. You know that 3D Secure is in place when you try to checkout, and end up being prompted to enter an OTP code or password. This extra authentication layer also enabled merchants to transfer liability for fraudulent transactions to the card issuer. 

3D Secure 1 was first rolled out in 2001, and though it has gained popularity as an effective tool to help reduce card fraud, it did have its own problems. Chief among the list of grievances against 3D Secure 1 is that the additional step required to complete the transaction didn’t mesh well with the payment flow, thus leading to a high cart abandonment rate. Secondly, lots of banks forced their customers to remember static passwords to complete 3D Secure authentication, and naturally, this didn’t work out too well.

Enter 3D Secure 2: Frictionless And Better Looking

3D Secure 2 aims to address these drawbacks while simultaneously strengthening security. One of the main features of 3D Secure 2 is the introduction of Risk Based Authentication (RBA) for transactions, thanks to its ability to support the sending of multiple data elements. The said data elements include payment-specific data such as shipping addresses, as well as contextual data, such as the customer’s device ID or previous transaction history.

The cardholder’s bank can then use this information to assess the risk level of the transaction and decide on an appropriate response to go along with it:

  • If the data is adequate for the bank to trust that the real cardholder is carrying out the purchase, the transaction goes through the “frictionless” flow and the authentication is completed without any additional input from the cardholder.
  • If the bank decides that it needs further proof, the transaction is sent through the “challenge” flow and the customer is asked to provide additional input to authenticate the payment.

Second, 3D Secure 1 was developed well before the rise of the smartphone. Today, we live our lives on our smartphones. As a result of the time it was built in, 3D Secure was very unpleasant to interact with unless you were in front of a PC. 3D Secure 1 would force a full page redirect, which was cumbersome and left customers potentially vulnerable to ‘Man-in-the-Middle’ attacks.

This has been rectified with 3D Secure 2, and banks can now offer a more seamless and less disruptive authentication experience. Instead of entering a password or waiting for a OTP-bearing text message to arrive, banks can now allow customers to authenticate the payment via fingerprint, face scanning, or even through the mobile banking app installed on their phone. 

3D Secure 2 has also been designed so that it is possible to embed the challenge flow directly within web and mobile checkout flows—without requiring full page redirects. This is a boon for any developer concerned with the user experience, like we are at Calcey. If a customer initiates an authentication on your site or webpage, the 3D Secure prompt now by default appears in a modal on the checkout page (browser flow).

3D Secure 1 left the user open to ‘Man-in-the-middle’ attacks / Credits: Unsplash

Issuers such as Visa and MasterCard have now made available mobile SDKs which make it easier to build ‘in-app’ authentication flows. Both processors have also made available UI guidelines for developers to help sidestep the problem of cart abandonment due to poor UI, which banks can be notorious for.

New age payment systems such as Apple Pay and Google Pay already support 3D Secure 2, and enabling these as payment options on your ecommerce site will allow you to quickly offer a seamless checkout and authentication experience.

While traditional banks may take some time to fully comply with SCA, payments processors such as Stripe and Braintree are already fully compliant. For instance, if you’re using Stripe to process payments, a quick upgrade of the Checkout integration is all you need to be fully compliant with 3D Secure 2.

Payment providers such as Stripe, Braintree, Square etc. are already SCA compliant / Credits: Unsplash

I run a small e-commerce startup? Should I worry about 3D Secure 2?

Not every online retailer needs to consider migrating to 3D Secure 2 immediately. If you are a small e-commerce site, you can temporarily postpone worrying about 3D Secure 2, since both 3D Secure 1 and 3D Secure 2 are expected to co-exist for some time. However, if your web analytics tools are telling you that you’re losing a lot of customers at the checkout stage due to 3D Secure 1, you may be better off considering an immediate shift to 3D Secure 2. While you’re at it, we would also recommend overhauling your backend infrastructure so that it is upgrade friendly, perhaps by integrating with Stripe and Shopify or something similar. This will free you from the headache of worrying about keeping your site’s code up-to-date, since these third party platforms will take care of everything for you. And if you need help, feel free to contact us.