Behold, big data! Here’s the what’s and the why’s.

what is big data

If you’re in charge of making business decisions, you’d no doubt understand that one of the most important things to running a company is visibility. Key decision makers have to ask themselves, how can I make the most profit in a predictable way? Sometimes, trying to figure out what our customers want can feel like staring into a dark void.

But it gets worse

Some bosses even see the internal workings of their business like an enigma. How efficiently are our assets running? How can we optimize our pricing? How do we improve customer service? It truly is amazing how many companies still manage to continue with such large question marks hanging over them.

Often, we don’t realize that the answers to our questions are right under our noses. If you think about it, all the little interactions that occur in our business operations actually emit little cues that one could use to learn more about their business. That is, if one knows not just where to look, but how to look!

That’s where big data comes in.

The concept of big data has come a long way from its status as a professional buzzword just some years ago. It refers to analyzing the copious amounts of information that businesses come into contact with, and generate daily. Today, it is well entrenched as a necessary business function.

Sure enough, business efforts have come to reflect that. Now, companies invest huge amounts of money to tackle big data issues such as:

  1. Capturing and converting the data – in fact, data capture methods have advanced so rapidly that it’s no longer a problem in itself. Today, humans generate roughly 2.5 quintillion bytes of data every day. Rather, businesses now have to deal with..
  2. Organizing and managing the data – merely storing and making the data easily accessible is a gargantuan task. Enterprise-level solutions have popped up from powerhouses like IDBS, spurring a data arms race from competitors.
  3. Analyzing and gaining insights from data – by far the most front and center in all big data problems today. Traditional computing and analytical programs are inadequate to effectively sift through and analyze the large amounts of data that is being generated daily. These days, techniques such as machine learning is used to make sense of big data.

What is big data used for?

The myriad ways to use big data reflect its general purpose nature. However, some that are most interesting to business owners are:

  1. Gaining deeper insights into customers, including their buying behavior. If taken even further, some companies can predict future customer behavior with great accuracy. Some might even be familiar with this fascinating (albeit creepy) case of Target successfully predicting when its customers were about to have a baby.
  2. Improving inventory management – bottlenecks can be quickly identified and optimized for efficiency. Algorithms can now examine past data for patterns and inform management on the best time to buy stock, where to locate it etc.
  3. Smart farming. Yup, even agriculture and farming is seeing huge benefits and use of big data. Coupled with cloud computing and the internet of things, decision making in farming can become increasing automated. The best part? Farmers can optimize for yield.
  4. Financial trading. Even finances wizzes will be shaking in their boots because algorithms can now comb masses of seemingly unrelated data and find causal relationships between them. This enables them to make predictions with scientific accuracy. Together with high frequency trading algorithms, these machines might kick human traders to the kerb with cups in their hands (I hope not, though!)


As a finance guy, I can’t pretend to be an expert on big data. But trust me, it is beyond a shadow of a doubt that big data is a force that will shake up how business is done forever. Many traditional jobs will be upended, and those that stay will have to learn to wield the power of big data and new technologies to even be in the game.