Big Data (Analyzed Right) can make you Omniscient (almost)

Picture all the data flowing over the web - structured data generated by business transactions, varied content on the millions of websites, sense and nonsense on the social media, log files recording site visits and visitor actions and data detected by all the sensors attached to machines. Big data analysis seeks to capture, cleanse, mine and analyze all these data, and generate meaningful insights.

Business Benefits from Big Data Analysis

Businesses could make a killing if they can identify consumer preferences as revealed by all the things we do online. We are already seeing ads that seem to be uncannily targeted at us. If we click an ad for a particular product, we see ads for that product popping up again and again while we browse the web. What happened was that the browser remembered our click and this click data was used by ad servers to serve ads based on our past actions.

Big data can help businesses improve their marketing, customer service, operational efficiencies and competitive advantage.

As to ourselves, the consumers, well, we stand to lose much of our privacy. What we do with each click, post, comment, purchase, etcetera will be "common knowledge" for the machines that do the things like showing ads to us!

Big Data Analysis:Processes and Tools

Advanced analytic tools capture, clean, analyze and interpret, and deliver useful information or predictions to decision makers. A big data project will broadly involve:

  • Clarifying business goals, what business outcome do you want?
  • Identifying the concrete actions that the business can take to achieve the outcome.
  • Determining the data sources, and how to collect data from these.
  • Capturing the data and storing these in databases that can accommodate all kinds of raw data.
  • Cleaning the data to eliminate useless junk.
  • Extracting the raw data, transforming them and loading these into formal databases.
  • Creating models that can make the data tell something about obtaining desired outcomes.
  • Presenting insights, information and predictions in a visual or other easily understood format.
  • Checking whether the model is working as intended in predicting and enabling desired outcomes.
  • Fine-tuning the model.

What are the tools that help in the analysis?

  • Hadoop: A software platform that can store and process massive amounts of data in a distributed fashion over a computer cluster. Data from sensors, website clicks and all kinds of sources can be imported into Hadoop in a raw format. This data can then be extracted, transformed and loaded into say, a relational database for easier processing.
  • Data Cleaners: Tools like NIFI can clean data before ingestion into the Hadoop files system There are also tools like OpenRefine that can cleanse huge volumes of messy data
  • Data Mining: Data mining can reveal patterns not known before and is critical to gain valuable insights from the mass of data. Tools like Mahout help mine the data in the Hadoop system
  • Data Analysis: Instead of revealing unknown patterns, data analysis involves asking specific questions and analyzing the data to find answers to these questions
  • Visualization: Information presented in the format of a mass of numbers takes time to understand. On the other hand, the same information presented in the forms of maps, charts and other visual forms can help quick understanding. Hadoop can work with common tools like MS Excel, or you can use specialized visualization tools like Tableau

As Ginni Rometty, CEO of IBM, says: “Many more decisions will be based on predictive elements versus gut instincts.” Use that as your selling point and sell data services to smaller businesses who cannot afford to hire high paid data scientists and analysts.

Internet of Things (IoT) Opens two Major Markets

Here is IoT in action. Sensors on a building's external walls sense the temperature outside. The information is communicated to actuators that set the cooling level of the air conditioner. In the room itself, sensors detect the presence of people and switch off the AC (and even lights) if no one is present. Result: Significant savings in energy.

Anything that can switched on and off can be controlled remotely in this new world of Internet. What you need are sensors to sense relevant parameters, connectivity to communicate the sensed data to a control center and actuators to adjust operations based on received communications. The layers involved are:

You may not be sure why your coffee pot should talk to your toaster, but precision technology powering an industrial Internet of Things has the potential to reshape the planet. To help clarify, Dr. Timothy Chou has created Precision to introduce us to the basics of the industrial Internet of Things (IoT).

  • Sensors that measure the conditions
  • Gateways that collect and transport data between sensors, users and applications
  • Networks that provide the connectivity to distribute the data as needed
  • Storage of all data for monitoring, analysis and management
  • Monitoring, Management and Analysis
  • What we see is a network of things all connected together.

The sensors could be in your farm measuring the moisture level of the soil with the water flow being automatically shut off if the level is adequate. Or they could be in the destination setting of your car sending info to your household equipment like the fridge (to set them on) and microwave (t start cooking the food inside) as soon as you start for home.

That means hackers can get into all those things and cause undesirable events to occur. In turn, this means that as IoT proliferates in the market, the network security market would also be growing to meet emerging new challenges.

Let us now have a glimpse at IoT applications by looking at the smart city of the future.

The Smart City Supported by IoT

Some major applications:

  • Environmental Monitoring: Air pollution, water quality, radiation and electromagnetic levels can all be monitored across the city and possible actions taken automatically to improve conditions or warn affected groups. Fires, radiation levels, approaching storms and much more can be spotted promptly and precautionary actions initiated in time.
  • Infrastructure: Smart roads that can not only monitor the condition of the roads but also generate electricity from the movement of vehicles and even people, improve the operation of self-driving cars and adjust street lighting levels. Smart waste bins measure and report their levels (which themselves can be controlled using solar compactors) and waste collection trucks sense these and collect only full or nearly full bin contents. Smart devices can also monitor railway tracks and bridges to report any emergencies.
  • Others: Smart grids connect energy consuming devices to the supply management systems helping the utility companies to balance loads and optimize energy usage. It will also enable consumers to remotely control their devices so that power bills are reduced. IoT systems can also sense water leakages and shut off supply, monitor traffic congestion and direct traffic to less congested roads, and help drivers find the best available parking spots instead of driving all over the place looking for parking spaces.

All of these applications create major benefits and create innumerable business opportunities. However, when everything is connected, hackers can make life intolerable by hacking into things and controlling them in undesirable ways. This is the major threat foreseen for Internet of Things and opens up the area of Security as another great business opportunity.

IoT in other Areas

IoT can not only create smart cities but also smarten up many other things. Businesses can benefit from lower cost and higher quality manufacturing, and automatically adjust production volumes depending on market feedback and procurement of supplies to these volumes. Healthcare can benefit from remote sensing of health conditions and emergencies, and even take over several nursing tasks such as helping patients get out of the bed.

The possibilities are innumerable. No wonder Internet of Things is considered a major growth area.


Back in the good old days of the PC, you got your computer. And loaded it with accounting, word processing, email and all other novelty stuff. And worked happily ever after doing wonderful things...

Frustrations were not long in appearing. The accounting software didn't meet your requirements. The email did not work when you travelled to another country. And the PC often worked too slow for your expectations.

Soon you began to see some great changes. Your email software was up on the Internet (the CLOUD) and you could receive and send mails from anywhere in the world. More applications moved to the cloud and your staff could work with these even while travelling all over.

You began to hear new words like SaaS, Software as a Service, that you could access from anywhere any time. Not only that, now you did not have to upgrade the software every time a new version appeared. Instead, the newest features were automatically available to you.


As things moved further, you could even hire computing facilities on the Net. Instead of spending a fortune on creating in-house capacities with expensive hardware and high paid personnel to operate and maintain these, you could enter into a contract with a cloud supplier who had the latest in everything and pay only for what you used.

PaaS, Platform as a Service, allowed you to develop applications with the latest tools paying only for what you used. IaaS, Infrastructure as a Service, provided the latest in hardware, Operating Systems and Security. Backup was automatic so you didn't have to worry about losing your irreplaceable data even if your staff forgot the boring task.

In essence, Cloud Computing involves:

  • Moving to the Internet instead of doing everything in-house with your own computers, software and maintenance staff
  • Paying only for what you needed instead of creating a high-cost computing infrastructure that might remain idle most of the time
  • Focusing on your business instead of worrying about your computers, operating systems, software needs, data backups and security needs
  • If you are worried about storing your sensitive data on systems owned by a third party and located in some mysterious corner of the world, you can go for a private cloud. After all, a cloud is just the Internet. So long as you can access your data and applications over the Internet, from any device from anywhere, it is cloud computing. 

So you can get a high capacity system that is connected to the Net (and properly secured). Store all your applications and data in it and keep it safe at your home while you work with smartphones, tablets, laptops and desktops from wherever you are. Just connect to the home system over the Internet and all the stuff is available to you.

Check the Four Terabyte Cloud Attached disk from Amazon to get an idea of what you can do with a personal cloud of your own!

Contact Us to explore cloud business options.