Data Analytics Process

Data Analytics Team includes Data Scientist, Data Analysts and Data Engineers who are involved in one or more Steps of the whole Data Analytics process.

The Data Analytics Process is divided into following Steps:-

  1. Collection of Data
  2. Integration of Data
  3. Preparation of Data
  4. Analytical Model Development
  5. Analytical Model Testing
  6. Analytical Model Revision
  7. Data Reporting / Visualization

Data Collection
Data Scientists identify the information they need for a particular analytics application and then work on their own or with data engineers and IT staffers to assemble it for use.


Data Integration
Data from different source systems may need to be combined via data integration routines, transformed into a common format and loaded into an analytics system such as Hadoop cluster, NoSQL database or data warehouse.


Data Preparation
Once the data that is needed is in place, the next step is to find and fix data quality problems that could affect the accuracy of analytics applications. That includes running data profiling and data cleansing jobs to make sure that the information in a data set is consistent and that errors and duplicate entries are eliminated. Additional data preparation work is then done to manipulate and organize data for the planned analytics use and data governance policies are applied to ensure that the data adheres to corporate standards and is being used properly.


Analytical Model Development
After Data Preparation, the data analytics work begins in earnest. A data scientist builds an analytical model using predictive modelling tools or other analytical software and programming languages such as Python, Scala, R and SQL.


Analytical Model Testing
In this phase, the model developed is initially run against a partial data set to test its accuracy.


Analytical Model Revision
Once the model is initially tested, it's revised and tested again, a process known as "training" the model that continues until it functions as intended. Finally, the model is run in production mode against the full data set, something that can be done once to address a specific information need or on an ongoing basis as the data is updated.


Data Reporting / Visualization
The last step in the data analytics process is communicating the results generated by analytical models to business executives and other end users to aid in their decision making.  That usually is done with the help of data visualization techniques, which analytics teams used to create charts and other infographics designed to male their findings easier to understand. Data visualizations are often incorporated into BI dashboard applications that display data on a single screen and can be updated in real time as new information becomes available.





Comments

Post a Comment

Popular posts from this blog

Hadoop Architecture version 1.x

Image
Hadoop Architecture is split into 3 categories:- Edge Node Name Node and Job Tracker Node Data Nodes and Task Tracker Nodes Client/Edge Node:   It is the interface between Hadoop cluster and outside network.For this reason, they are sometimes referred as gateway nodes. Client applications talk to the Name Node whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The Name Node responds the successful requests by returning a list of relevant Data Node servers where the data lives. Usage They are often used as staging areas for data being transferred into the Hadoop cluster.  Most commonly, edge nodes are used to run client applications and cluster administration tools. Tools like Oozie, Pig, Sqoop, and management tools such as Hue and Ambari run well there.  Client Node directly controls Data Node based on information. Master/Slave:   Here, we have a concept of Master...
Read more

Hadoop Components

Components of Hadoop File System HDFS (Primary Data storage system) Amazon S3 Azure Execution Engine MapReduce - launched since Hadoop version 1 Apache Spark - launched since Hadoop version 2 (Available as part of Hortonworks and Cloudera distributions) Apache Tez - launched since Hadoop version 2 (Available as part of Hortonworks distribution but not in Cloudera) Ecosystem Tools Hive - Datawarehousing tool on HDFS (used primarily for Structured Data and to some extent for Semi-Structured Data) Pig - ETL tool where source is other system and target is HDFS (used for Structured, Semi-Structured and Unstructured Data) Sqoop - Tool for Importing/Exporting data with relational database like Oracle (used for Structured Data) Oozie - Tool for monitoring Workflows and Scheduling Jobs Zookeeper - Configuration Manager
Read more

What is Data Analytics?

Data Analytics is the process of examining data sets in order to draw conclusions about the information they contain, increasingly with the aid of specialized systems and software. Data analytics technologies and techniques are widely used in commercial industries to make more informed business decisions and by scientists and researchers to verify or disprove scientific models, theories and hypotheses. Data Analytics refers to an assortment of applications from basic business intelligence (BI), reporting and online analytical processing (OLAP) to various forms of advanced analytics. In that sense, it is similar to business analytics, another umbrella term for approaches to analyzing data - with the difference that the latter is oriented to business uses, while data analytics has a broader focus. Advantages of Data Analytics The advantages of Data Analytics are in boosting business performance by:- Increasing Business Revenues Improve Operational Efficiency Optimize Marketing...
Read more