What is big data?
Big data is a field that:
treats ways to analyze,
systematically extract information from,
or otherwise deal with data sets
That is too large or complex to be dealt with by traditional data-processing application software. Data with many cases (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy and data source.
Big data was originally associated with three key concepts: volume, variety, and velocity. Later another two concepts were added to define it which are: value and veracity.
When we handle big data, we may not sample but simply observe and track what happens. Therefore, big data often includes data with sizes that exceed the capacity of traditional software to process within an acceptable time and value.
Current usage of the term big data tends to refer to the use of predictive analytics, user behavior analytics, or certain other advanced data analytics methods that extract value from data, and seldom to a particular size of data set.
The thing with Big Data is that it has to have enough volume so that the amount of bad data or missing data becomes statistically insignificant. When the errors in the data are common enough to cancel each other out, when the missing data is proportionally small enough to be negligible and when the data access requirements and algorithms are functional even with incomplete and inaccurate data, then we have "Big Data".
Artificial intelligence (AI), mobile, social and the Internet of Things (IoT) are driving data complexity through new forms and sources of data. For example, big data comes from sensors, devices, video/audio, networks, log files, transactional applications, web, and social media — much of it generated in real time and at a very large scale.
It is said that Big Data implies a large amount of information (terabytes and petabytes or zettabytes) which is true to some extent but Big Data is not really about the volume, it is about the characteristics of the data.
What is Big Data Analytics?
Big data analytics is the use of advanced analytic techniques against very large, diverse data sets that include structured, semi-structured and unstructured data, from different sources, and in different sizes from terabytes to zettabytes.
It is a process used to extract meaningful insights, such as hidden patterns, unknown correlations, market trends, and customer preferences.
Analysis of big data allows analysts, researchers and business users to make better and faster decisions using data that was previously inaccessible or unusable. Businesses can use advanced analytics techniques such as text analytics, machine learning, predictive analytics, data mining, statistics and natural language processing to gain new insights from previously untapped data sources independently or together with existing enterprise data.
History and evolution
The concept of big data is not recent, in fact it has been around for years; most organizations now understand that if they capture all the data that streams into their businesses, they can apply analytics and get significant value from it. But even in the 1950s, decades before anyone uttered the term “big data,” businesses were using basic analytics (essentially numbers in a spread sheet that were manually examined) to uncover insights and trends.
The new benefits that big data analytics brings to the table, however, are speed and efficiency. Whereas a few years ago a business would have gathered information, run analytics and unearthed information that could be used for future decisions, today that business can identify insights for immediate decisions. The ability to work faster – and stay agile – gives organizations a competitive edge they didn’t have before.
Why Big Data Analytics?
Big Data analytics is fuelling everything we do online—in every industry.
Take the music streaming platform Spotify for example. The company has nearly 96 million users that generate a tremendous amount of data every day. Through this information, the cloud-based platform automatically generates suggested songs—through a smart recommendation engine—based on likes, shares, search history, and more. What enables this are the techniques, tools, and frameworks that are a result of Big Data analytics.
If you are a Spotify user, then you must have come across the top recommendation section, which is based on your likes, past history, and other things. It is done by utilizing a recommendation engine that leverages data filtering tools which collects data and then filters it using algorithms. This is what Spotify does.
Big Data analytics provides various advantages—it can be used for better decision making, preventing fraudulent activities, among other things.
Big data analytics helps organizations harness their data and use it to identify new opportunities. That, in turn, leads to smarter business moves, more efficient operations, higher profits and happier customers. In his report Big Data in Big Companies, IIA Director of Research Tom Davenport interviewed more than 50 businesses to understand how they used big data. He found they got value in the following ways:
1. Cost reduction. Big data technologies such as Hadoop and cloud-based analytics bring significant cost advantages when it comes to storing large amounts of data – plus they can identify more efficient ways of doing business.
2. Faster, better decision making. With the speed of Hadoop and in-memory analytics, combined with the ability to analyse new sources of data, businesses are able to analyse information immediately – and make decisions based on what they’ve learned.
3. New products and services. With the ability to gauge customer needs and satisfaction through analytics comes the power to give customers what they want. Davenport points out that with big data analytics, more companies are creating new products to meet customers’ needs.
Tools and Techniques
The following are the techniques used in big data analytics:
1. Association rule learning
Are people who purchase tea more or less likely to purchase carbonated drinks?
Association rule learning is a method for discovering interesting correlations between variables in large databases. It was first used by major supermarket chains to discover interesting relations between products, using data from supermarket point-of-sale (POS) systems.
Association rule learning is being used to help:
place products in better proximity to each other in order to increase sales
extract information about visitors to websites from web server logs
analyse biological data to uncover new relationships
monitor system logs to detect intruders and malicious activity
identify if people who buy milk and butter are more likely to buy diapers
2. Classification tree analysis
Which categories does this document belong to?
Statistical classification is a method of identifying categories that a new observation belongs to. It requires a training set of correctly identified observations – historical data in other words. The output is in the tree format with nodes and there is an association in the nodes. And this can be read to form if-then rules. It builds classification in the form of trees structure. As shown in figure below, it divides the data nodes into small subgroup. These methods can be used when the data mining task has predictions or classification of outcomes.
Statistical classification is being used to:
automatically assign documents to categories
categorize organisms into groupings
develop profiles of students who take online courses
3. Genetic algorithms
Which TV programs should we broadcast, and in what time slot, to maximize our ratings?
Genetic algorithms are inspired by the way evolution works – that is, through mechanisms such as inheritance, mutation and natural selection. These mechanisms are used to “evolve” useful solutions to problems that require optimization.
Genetic algorithms are being used to:
schedule doctors for hospital emergency rooms
return combinations of the optimal materials and engineering practices required to develop fuel-efficient cars
generate “artificially creative” content such as puns and jokes
4. Machine Learning
Which movies from our catalogue would this customer most likely want to watch next, based on their viewing history?
Machine learning includes software that can learn from data. It gives computers the ability to learn without being explicitly programmed, and is focused on making predictions based on known properties learned from sets of “training data.”
Machine learning is being used to help:
distinguish between spam and non-spam email messages
learn user preferences and make recommendations based on this information
determine the best content for engaging prospective customers
determine the probability of winning a case, and setting legal billing rates
5. Regression Analysis
How does your age affect the kind of car you buy?
At a basic level, regression analysis involves manipulating some independent variable (i.e. background music) to see how it influences a dependent variable (i.e. time spent in store). It describes how the value of a dependent variable changes when the independent variable is varied. It works best with continuous quantitative data like weight, speed or age.
Regression analysis is being used to determine how:
levels of customer satisfaction affect customer loyalty
the number of supports calls received may be influenced by the weather forecast given the previous day
neighborhood and size affect the listing price of houses
to find the love of your life via online dating sites
6. Sentiment Analysis
How well the new return policy is received?
Sentiment analysis helps researchers determine the sentiments of speakers or writers with respect to a topic.
Sentiment analysis is being used to help:
improve service at a hotel chain by analyzing guest comments
customize incentives and services to address what customers are really asking for
determine what consumers really think based on opinions from social media
7. Social Network Analysis
How many degrees of separation are you from Kevin Bacon?
Social network analysis is a technique that was first used in the telecommunications industry, and then quickly adopted by sociologists to study interpersonal relationships. It is now being applied to analyze the relationships between people in many fields and commercial activities. Nodes represent individuals within a network, while ties represent the relationships between the individuals.
Social network analysis is being used to:
see how people from different populations form ties with outsiders
find the importance or influence of a particular individual within a group
find the minimum number of direct ties required to connect two individuals
understand the social structure of a customer base
The following are the tools used in big data analytics:
1. Hadoop
An open-source framework, Hadoop offers massive storage for all kinds of data. With its amazing processing power and capability to handle innumerable tasks, Hadoop never allows you to ponder over hardware failure. Though you need to know Java to work with Hadoop, it’s worth every effort.
Apache Hadoop is the one of the technology designed to process Big Data, which is unification of structured and unstructured data of huge volume. Apache Hadoop is an open source platform and processing framework that exclusively provides batch processing. Hadoop was firstly influenced by Google's Map Reduce. In Map Reduce software framework the whole program is divided into a number of parts which are small in size. These small parts are also called as fragments. These fragments can be executed on any system in the cluster.
Components of Hadoop: There are a lot of components which are used in composition of Hadoop. These all worked together to execute batch data.
Main components are as:
HDFS: The Hadoop Distributed File System (HDFS) is the main component of the Hadoop software framework. It is the file system of Hadoop. HDFS is configured to save large volume of data. It is a fault –tolerant storage system that stores large size files from TB to PB. There are two types of nodes in HDFS Name node and Data node. Name Node, works as the master node. It contains all the information related to the entire data node. It has the information of free space, addresses of nodes, and all the data that they store, active node, passive node. It also keeps the information of task tracker and job tracker. Data node is also known as slave node. Data node in Hadoop is used to store the data. And it is the duty of TaskTracker to keep the track of on-going job which resides on the data node and it also take care of the jobs coming from name node.
MapReduce: It is a framework that helps developers to jot down programs to method massive volume of unstructured knowledge parallel over a distributed design. MapReduce consists of many elements like JobTracker, TaskTracker and JobHistorySever etc. It is additionally referred to as the Hadoop's native instruction execution engine. It was introduced to process the huge amount of data and to store these huge data on commodity hardware. For processing the large volume data it uses clusters to store records. Map function and Reduce function are two functions that are the base of the Map Reduce programming model. In master node the Map function works. And it accepts the input. And after then divide that accepted input into sub modules and then distribute it into slave nodes.
YARN (Yet another Resource Negotiator): It is the core Hadoop services that supports two major Services: World resource management (ResourceManager) and per application management (ApplicationMaster). It is the cluster coordinating element of the Hadoop stack. YARN makes it attainable to execute.
It is the MapReduce engine that is responsible for practicality of Hadoop. MapReduce is a framework that run on hardware that are less costly. It doesn't conceive to save anything in memory. MapReduce has unimaginable measurability potential. It has been employed in creation of thousands of nodes. Different additions to the Hadoop scheme will scale back the effect of this to variable degrees; however it will always be an element in quickly implementation of an inspiration on a cluster of Hadoop.
Working of Hadoop:
In the architecture of Hadoop there is only one master node, works as master server known as JobTracker. There are several slave node servers known as TaskTracker's. Keeping the track of the slave nodes is the central job of JobTracker. It established an interface infrastructure for various job. Users input the MR (MapReduce) jobs to the JobTracker, where the pending jobs are reside in queue. The order of access is FIFO. It is the responsibility of JobTracker to coordinate the mapper’s execution and reducer’s execution. When the Map Task is completed, the JobTracker starts its functionality by initiating the reduce task. Now it is the duty of JobTracker to give proper instruction to TaskTracker. After then TaskTracker starts the downloading files and mainly concatenate the various files into a single unit (entity).
2. Spark
Apache Spark is a great open source option for people who need big data analysis on a budget. This data analytics engine's speed and scalability have made it very popular among data scientists. One of the great things about Spark is how compatible it is with almost everything. It can also be used for a variety of different things, like cleansing and transforming data, building models for evaluation and scoring, and determining data science pipelines for production. The lazy execution is really nice. This feature allows you to set a series of complex transformations and have them represent as a single object. This allows you to inspect its structure and end result without executing individual steps along the way. Spark even checks for errors in the execution plan before submitting it, which prevents bad code from taking over the process.
3. MongoDB
MongoDB is a contemporary alternative to databases. MongoDB is a database that is based on JSON documents. It is written in C++ and launched in 2009, and is still expanding.
It’s the best for working on data sets that vary or change frequently or the ones that are semi or unstructured. . MongoDB database basically holds the set of data that has no defined schema. There is no predefined format like tables, and can stores data in the form of BSON documents. BSON are binary encoded JSON like objects. User can go for MongoDB rather than MySQL if the requirement is knowledge intensive because it stores information and queries.
MongoDB is especially engineered for storage of information and retrieval of stored information. It can do processing and measurability. It supported C++ and belongs to the NoSQL family. Some of the best uses of MongoDB include storage of data from mobile apps, content management systems, product catalogues and more. Like Hadoop, you can’t get started with MongoDB instantly. You need to learn the tool from scratch and be aware of working on queries.
4. Tableau
Tableau is extremely powerful. The fact that it is one of the most mature and powerful options available shows as soon as you see the available features. It’s a bit steeper to learn this platform, but once you do it is well worth it.
Tableau has been around since the early days of big data analytics, and it continues to mature and grow with the industry. It is extremely intuitive and offers comprehensive features. Tableau can handle any amount of data, no matter the size. It offers customizable dashboards and real time data in visual methods for exploration and analysis.
It can blend data in powerful ways because of how flexible the settings are. It has tons of smart features and works at lightning speed. Best of all, it is interactive and can even work on mobile devices and share data through shared dashboards.
5. Elastisearch
This open-sourced enterprise search engine is developed on Java and released under the license of Apache. It works across multiple platforms and can distribute data easily. It even has a Lucerne based search engine. This is one of the most popular enterprise search engines on the market today.
One of its best functionalities lies in supporting data discovery apps with its super-fast search capabilities.
Elastisearch is included as an integrated solution with Logstash and Kibana. Logstash can collect data and parse logs and Kibana is a great platform for the visualization of data and analysis. The three products work together in what is known as Elastic Stack. A lot of people don’t like open source software because it can be difficult to follow when there’s no one to call for tech support. Happily, Elastic has a very active community and their documentation is incredibly easy to understand. It makes it super easy to use the NO-SQL search engine and storage. Elastic also has APIs for just about anything you will ever need.
6. Cassandra
Used by industry players like Cisco, Netflix, Twitter and more, it was first developed by the social media giant Facebook as a NoSQL solution.
It’s a distributed database that is high-performing and deployed to handle mass chunks of data on commodity servers. Apache’s Cassandra offers no space for failure anywhere in the awesome set of features, which include simple ring architecture, automated replication, and easy log structured storage and is one of the most reliable Big Data tools. Although troubleshooting and maintenance can take a little more effort than other tools, the free price makes it worth it. Mix that with the fact that it has such a rapid response time, and doesn’t take too much system resources.
7. Drill
It’s an open-source framework that allows experts to work on interactive analyses of large scale datasets. Developed by Apache, Drill was designed to scale 10,000+ servers and processes in seconds data of size of petabytes and millions of records. It supports tons of file systems and databases such as MongoDB, HDFS, Amazon S3, Google Cloud Storage and more.
8. Oozie
One of the best workflow processing systems, Oozie allows you to define a diverse range of jobs written or programmed across multiple languages. Moreover, the tool also links them to each other and conveniently allows users to mention dependencies.
9. Apache Storm
Storm supports real-time processing of unstructured data sets. It is reliable, fault-proof and is compatible with any programming language. Hailing from the Apache family of tools, Twitter now owns Storm as an open-sourced real-time distributed computing framework.
10. Kafka
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
It is a distributed streaming platform that is used for fault-tolerant storage.
11. HCatalog
HCatalog allows users to view data stored across all Hadoop clusters and even allows them to use tools like Hive and Pig for data processing, without having to know where the datasets are physically present. A metadata management tool, HCatalog also functions as a sharing service for Apache Hadoop.
Codersarts is a top rated website for students which is looking for online Programming Assignment Help, Homework help, Coursework Help in deep learning to students at all levels whether it is school, college and university level Coursework Help or Real time project.
Hire us and Get your projects done by computer science expert. CONTACT US NOW
Comments