How to

How to set up Kafka in a Docker container

At Calcey, we recently found ourselves having to deal with linking a legacy system with a new information system on behalf of a client. In order to avoid complications, we explored the possibility of deploying Kafka within a docker container.

What is Kafka?

Kafka is an open-source, fault-tolerant event streaming platform. Kafka can help bridge the information gap between legacy systems and newer systems. Imagine a situation where you have a newer, better system that needs data from an older, legacy system. Kafka can fetch this data on behalf of the developer without the need to build an actual connection between the two systems.

Kafka, therefore, will behave as an intermediary layer between the two systems.

In order to speed things up, we recommend using a ‘Docker container’ to deploy Kafka. For the uninitiated, a ‘Docker container’ is a lightweight, standalone, executable packages of software that include everything needed to run an application: code, runtime, system tools, system libraries, and settings.

To deploy Kafka, three pieces of the puzzle need to fall in place: the ZooKeeper server, Kafka server, and a connector to the data source. In addition, we will be making use of SQL server’s Change Data Capture (CDC) feature to feed data into Kafka. CDC records any insertion, updating, and deletion activity that is applied to a SQL Server table. This makes the details of the changes available in an easily consumed relational format. But that’s a topic for another day.

The easiest way to set all this up is to use Debezium. We recommend using the Debezium image which can be downloaded from This image will allow you to configure ZooKeeper, Kafka and the Connector in one go.

With both ZooKeeper and Kafka now set up, all you have to do is tell Kafka where your data is located. To do so, you can connect Kafka to a data source by means of a ‘connector’. While there is a wide range of connectors available to choose from, we opted to use the SQLServer connector image created by Debezium. Once a connection is established with the data source, pointing the connector back to the Kafka server will ensure that all changes are persisted with.

And that’s all there is to deploying Kafka in a Docker Container!


Lessons for startups from Zoom

Zoom, the video conferencing startup which managed to beat Cisco’s WebEx at its own game, recently went public. Leaving the IPO aside, there was a lot of media attention on Zoom’s history as a company, since it very much broke the stereotype of the ‘hot Silicon Valley startup’.

Before Zoom arrived on the scene, many thought that the problem of video conferencing had been solved thanks to Cisco’s WebEx and Skype. But that’s not what Eric Yuan thought. A founding engineer on the WebEx team, Eric was passionate about building a video conferencing solution that just worked. He tried to implement his ideas at WebEx, but his bosses didn’t want to listen, and Eric left WebEx to found Zoom.

Eric Yuan, founder of Zoom / Source: Thrive Global

Having looked at Zoom’s growth from afar, here’s what we think all other startups can learn from Zoom

Be focused on the product, maniacally

This story about how focused Zoom is on improving its product comes directly from Sequoia Capital, one of Zoom’s investors. But before they became an investor, Sequoia was a paying customer of Zoom.

“When Sequoia first reached out to Eric in 2014, he told us he admired our work but wasn’t looking for funding. Then he asked for an intro to our IT department, to see if they’d be interested in using Zoom. He cared more about our business than he did about our money — because he was, as he is today, singularly focused on his mission of making people happy.”

-Carl Eschenbach & Pat Grady, Sequoia Capital

Many early-stage startups suffer from a tendency to focus on securing funding, instead of focusing on their product and acquiring paying customers. But Zoom’s approach of focusing on acquiring paying customers, which indirectly gave them more leverage when negotiating with investors later.

To exhibit how focused Zoom is on making its product good, consider this. In a recent feature on Zoom, Forbes columnist Alex Conrad wrote that Zoom could operate well even on a connection with 40% packet loss, which is a boon for those on spotty or slow connections.

Zoom’s platform / Source: Company S-1

Build sustainable revenue streams

In Silicon Valley, there is a tendency to chase revenue growth which is usually fuelled by deep discounts and/or by running at a loss. A ready example can be found in the meal delivery startup sector, where profitability remains elusive yet discounts, plentiful. Essentially, most startups in the sector are hemorrhaging money to make a little bit of money or no money at all. Worse yet, some will never see a cent in profits for a very, very long time. Not so with Zoom.

Consider the following, taken from the second page of Zoom’s S-1 document:

“Our revenue was $60.8 million, $151.5 million and $330.5 million for the fiscal years ended January 31, 2017, 2018 and 2019, respectively, representing annual revenue growth of 149% and 118% in fiscal 2018 and fiscal 2019, respectively.”

But the next section makes things even more interesting:

“We had a net loss of $0.0 million and $3.8 million for the fiscal years ended January 31, 2017, and 2018, respectively, and net income of $7.6 million for the fiscal year ended January 31, 2019.”

Simply put, Zoom was already a profitable company when it sought to list its shares, a rare achievement in the startup world. For comparison, look at the finances of some other startups which went public in recent times:

  • Pinterest, who filed on March 22nd, the same day as Zoom, made $755M in revenue in the fiscal year 2018 but a net loss of $63M.
  • PagerDuty, who filed on March 16th, made $79.6M revenue in the fiscal year 2018, but a net loss of $38.1M.
  • Lyft, who filed on March 1st, made $2.2B revenue in the fiscal year 2019, but a net loss of $911.3M.

In the technology world, running at a loss in order to get a shot at an IPO is widely considered a necessary evil. But Zoom was comfortably in the black, which allowed the company to list at a valuation of USD 8.98 billion.

Zoom’s financials remain healthy / Source: Forbes

Your users can be your best evangelists

Zoom credits its growth to its bottom-up user generation cycle, which conceptually, shares a few similarities with Dropbox’s famous referral system. With Zoom, users can sign up and invite others to a meeting (for free) and when they realize how easy-to-use and great the product is, they sign up too and then pay for more features.

Zoom’s S-1 states that amongst others, the company had 344 customers who generated more than $100K in annual revenue, up 141% YoY. This customer segment accounted for 30% of Zoom’s revenues in FY’19. 55% of those 344 customers started with at least one free host prior to subscribing. As more and more customers invite people to meetings held on Zoom, those numbers are only going to rise. Consider this quote from a Sequoia spokesperson:

“We had been watching the video conferencing space for many years because we were convinced that it was a huge market primed for innovation. But we couldn’t find a product that customers loved to use. That changed when our portfolio companies started raving to us about Zoom.”

Execution matters

When Eric Yuan decided to build Zoom, the problem of video conferencing was, for all intents and purposes, considered to be solved. There were many incumbents, ranging from WebEx to Skype and Google Hangouts. But they were full of problems. Some were built for an age where video conferencing was done in front of a computer, some didn’t have features such as file sharing from mobile, etc. In trying to build a better video conferencing product that truly lived off the cloud, and scaled simply and scaled well, Zoom did not try to reinvent the wheel. Instead, they just set out to make a motorized car while the rest of the world was content to ride on horse-drawn carriages. Unsurprisingly, Zoom is a company favoured by Social Capital CEO Chamath Palihapitiya, who ranks it on the same level as Slack, another successful tech startup (of which Palihapitiya is an investor).

If you’re building a startup yourself, we highly recommend that you keep an eye on Eric and his team. In the meantime, if you are a user of Zoom, what was your experience with the product like? Do you think Zoom will become the next Slack? Let us know in the comments!



Reactive Programming: How Did We Get Here?

In a world that continues to evolve rapidly, the way we build software too is in a constant state of flux. The heavy architectures of yesterday have given way to new, lighter, more agile architectures such as reactive programming.

What Is Reactive Programming?

At its core, reactive programming is a way of defining inter-communications and codependent behaviors of program components, so that there is minimal interdependence..

In simple terms, this is achieved by each individual component exposing data about changes happening within them in a format and medium accessible to others, while allowing other components to act upon this data if it is of any relevance to them.

In today’s always-on, completely mobile world users expect responses in milliseconds, along with 100% uptime. Only systems that are responsive, resilient, elastic and message driven can deliver this kind of performance, which is why they can be termed ‘reactive systems’.  And in order to build reactive systems, we must employ ‘reactive programming’.

How Did Reactive Programming Come To Be?

As a technique, reactive programming has been in existence since the seventies (and perhaps even before) and is not something that rose to prominence recently. For instance, when the Graphical User Interface (GUI) was first introduced, reactive programming techniques could have been used to reflect changes in the mouse pointer’s position on the screen.

Examples of Reactive Programming At Work

In general, reactive programming can be seen in action in the following instances:

  • External Service Calls
    Under reactive techniques, a developer will be able to optimize the execution of any external HTTP or REST calls, thus benefiting from the promise of ‘composability’ offered by reactive programming.
  • Highly Concurrent Message Consumers
    Reactive programming can also be used in situations where messages need to be processed between multiple systems, a need that frequently arises in the enterprise space. The patterns of reactive programming are a perfect fit for message processing since events usually translate well into a message.
  • Spreadsheets
    Often the favourite tool (or bane) of many cubicle dwellers, Excel is another perfect example of reactive programming at play. Think of a scenario where you built a model with interdependencies between several cells. A group of cells will be linked to one cell, or even another spreadsheet. Making a change in the precedent cell, will automatically force changes in the dependent cells. This is in effect, reactive programming at play.

When To Use Reactive Programming

In practice, programmers use both reactive and traditional techniques. As such, there is no definitive guide on when to use reactive programming and when not to. It’s more of an intuitive understanding, which a developer will gain over time through experience and countless hours of coding.

As a rule of thumb, if an application’s architecture is simple and straightforward, a developer may be better served by sticking to traditional code structuring methods. Breaking this rule may leave you with an over-engineered product on your hands, which is undesirable.

But, as with all things, proceed with caution. If you do end up following a reactive programming technique over imperative or some other technique, you will essentially be accepting a much higher level of code complexity, in return for more flexibility and robustness in the components of your program. Therefore, it is upto you to weigh the costs against the benefits.

The reactive landscape has evolved so much that today, we have Reactive UIs, ReactiveX APIs, and even Reactive microservices. Overall, these developments point towards a very bright future for reactive programming as a practice.

That wraps up our thoughts on the evolution of reactive programming.

What would you like to see us discuss next? Let us know in the comments below.