Gaurav Bhalotia, udaan Speaks on Data Storage and Infrastructure

Gaurav Shares

The challenge in today’s world with data is that people are on two ends of the spectrum, there will be some products/ companies that say I don’t have data for my use case. In my previous start-up, we were in that bucket, how do we get data for the application that I want to build? There are a bunch of people on the other end who have lots of data. Having data is not the challenge, making sense of the data is the bigger challenge. For example, there are CCTV cameras today that are generating a lot of data, a humongous amount of data that we cant process. I think that was the genesis of the data platform – can we centralize it, make sense of the data, make it easy to process and extract value out of it. I think we were not the ones who started that trend, US companies like Facebook showed that there has to be a data platform and we were the first to adopt it in India. Flipkart very quickly realized that data would become a differentiator. It is not because Flipkart has been around for a long time that it will win but it is because Flipkart has understood what these users want, what works, what doesn’t work and how we create a better experience for the users day in and day out. We started that, in today’s time I will be surprised if any startup says that they don’t have a data platform, data is the key. We are discussing hybrid cloud vs single cloud, so yes this is a question that is on the top of the mind for a lot of CTOs and CIOs, I hear a lot about this question in other forums too. We are at a stage in the evolution of technology, cloud came in and we realised that we can focus on what matters to the company and build our application, our functionality and offload availability, scale and reliability to a cloud provider. I still remember in 2012 we had a massive catalogue in Flipkart which powered our website, we were storing it in MySQL. There was no relational database service or cloud providers who did that. So, there was a large dedicated team that ensured we could run the data at that scale and keep it up. There were investments in specialized hardware to store this data. It is a deep, highly specialized, skill-oriented work, hard to get people who do this. You need to be on it day in and day out. Now cloud comes in, so startups on day one can go to RDS, you start small but you can quickly scale to 1000 requests in a minute, it can support that with all the data coming in. So, the way cloud started off, people were really overjoyed with it solving their problems, businesses could focus on what they needed. Generally, clouds are reliable, they have a concept of SLAs – what percentage of time can one say the cloud will be up and available. We measure them in nine’s – 4 nines, 5 nines and so on. These tell you, in a month what is the maximum downtime that can happen, we are looking at most 10 minutes of downtime in a month. What this did was now most of the companies are on cloud today and sorts of applications are coming on cloud and this is where this question comes – now that you are on the cloud, are there any challenges or downsides to being here? I believe the biggest thing you end up seeing eventually for a company which grows massively like udaan has a huge footprint on the cloud. Even at Flipkart, we had a huge footprint on our own data centres. You begin to realise that if things go wrong you have very limited control because someone else is doing it. They are not expected to go down, if they do go down something really bad has happened and anything can go wrong. For instance, if a company’s data centre is in a country that is prone to flooding, it would be just water, which is out of your control. If things go wrong, it is usually bad, most of us face that challenge – we have to figure out a way to mitigate the downside or impact on the end customers. There can be instances where it is very little you can do, the provider is fixing the issue. This is one kind of challenge you see, second challenge is that there are multiple cloud providers, some are good at something, others at something else. As the dependency on cloud platform services starts increasing, you realize some platforms are much better than others on some of these services. The third challenge is you see that the cost starts becoming material over time. Like everything in life what is good eventually ends up showing the challenges on the bad side. What is working to your advantage on the cloud, for example, is to empower my engineering team. You need to build a new service and scale it out and it is just a command or a click on the UI. What it also means is that at scale you need the right governance otherwise your cost will just blow it up. So, that’s the third thing that hits companies. We’ve seen that at scale most companies have gone to their own cloud strategies, however, there have been exceptions. All the large players like Facebook, Google and others started depending on cloud providers. However, Google eventually externalised it like GCP is available as a platform to others. And then there are companies like Netflix who decided that they will not have their own cloud, it continued to depend on AWS. But they have very deep connections, so there is a lot of custom work happening for them to solve these three challenges – cost, availability of features, capabilities and uptimes just to make sure they have the least number of challenges. A lot of other companies like Alibaba in china ended up building a cloud for themselves. Flipkart built its own cloud, we had a cloud team. My take on this is a multi-cloud or a hybrid cloud strategy, that is one of the ways to mitigate it. You create infrastructure which allows you to take the best of what is available to you or certain services. For example, you depend on cloud API to do OCRs or to do accurate facial recognition and one of the players has a much better performance at this time. Why wouldn’t you do what is best for your business? So the availability to go to multiple clouds – I would say the availability to have something of your own which allows you to do deeper integrations with your data. For example, if you are storing sensitive data like Adhaar card for KYC, you require native support like hashing and encryption, ensuring the data is more secure. You don’t want to put this data on the cloud, keeping these options is the thing to do. But these require investments and do not come for free. One of the biggest costs of the cloud is to take the data out. If you are on two providers, you’ll have to figure out how to bring the data out.

Watch the full podcast here:

JTNDaWZyYW1lJTIwd2lkdGglM0QlMjI1NjAlMjIlMjBoZWlnaHQlM0QlMjIzMTUlMjIlMjBzcmMlM0QlMjJodHRwcyUzQSUyRiUyRnd3dy55b3V0dWJlLmNvbSUyRmVtYmVkJTJGbUdmRHJzUXRfQ00lMjIlMjB0aXRsZSUzRCUyMllvdVR1YmUlMjB2aWRlbyUyMHBsYXllciUyMiUyMGZyYW1lYm9yZGVyJTNEJTIyMCUyMiUyMGFsbG93JTNEJTIyYWNjZWxlcm9tZXRlciUzQiUyMGF1dG9wbGF5JTNCJTIwY2xpcGJvYXJkLXdyaXRlJTNCJTIwZW5jcnlwdGVkLW1lZGlhJTNCJTIwZ3lyb3Njb3BlJTNCJTIwcGljdHVyZS1pbi1waWN0dXJlJTIyJTIwYWxsb3dmdWxsc2NyZWVuJTNFJTNDJTJGaWZyYW1lJTNF