Big Data Analytics - Coursework Assignment

Introduction

In this blog post, we will see into the intricacies of a significant project titled "Big Data Analytics - Coursework." This project revolves around understanding, analyzing, and deriving insights from the UNSW-NB15 dataset, which encompasses a rich collection of network traffic data designed for cybersecurity analysis.

Project Overview:

The project aims to conduct a comprehensive analysis of the UNSW-NB15 dataset, which encompasses a blend of real-world normal activities and synthetic contemporary attack behaviors. The dataset, generated using the IXIA PerfectStorm tool within the Cyber Range Lab of the Australian Centre for Cyber Security (ACCS), is designed for big data analytics. The tasks involve understanding the dataset, querying and analyzing using Apache Hive, performing advanced analytics using PySpark, and documenting the entire process.

Tasks:

1 . Understanding Dataset: UNSW-NB15

The UNSW-NB151 dataset's raw network packets were generated using the IXIA PerfectStorm tool within the Cyber Range Lab of the Australian Centre for Cyber Security (ACCS). This dataset was designed to combine real modern normal activities with synthetic contemporary attack behaviors. Tcpdump was employed to capture 100 GB of raw traffic, resulting in Pcap files. The dataset encompasses nine types of attacks: Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shellcode, and Worms. To further analyze the data, Argus and Bro-IDS tools were utilized, and twelve algorithms were developed, generating a total of 49 features along with their corresponding class labels.

a). The features are outlined in this section.

b). The quantity of attaks and their respective sub-categories is delineated here.

c). In this coursework, we utilize a total of 10 million records stored in a CSV file (available for download). The file size amounts to approximately 600MB, which is sufficiently large to warrant the application of big data methodologies for analysis. As a specialist in big data, our initial approach involves comprehending the dataset's features before implementing modeling techniques. Should you wish to view a subset of this dataset, you can import it into Hadoop HDFS and execute a Hive query to display the first 5-10 records for better comprehension.

Dataset Features : -

No.	Name	Type	Description
1	srcip	nominal	Source IP address
2	sport	integer	Source port number
3	dstip	nominal	Destination IP address
4	dsport	integer	Destination port number
5	proto	nominal	Transaction protocol
6	state	nominal	Indicates the state and its dependent protocol
7	dur	Float	Record total duration
8	sbytes	Integer	Source to destination transaction bytes
9	dbytes	Integer	Destination to source transaction bytes
10	sttl	Integer	Source to destination time to live value
11	dttl	Integer	Destination to source time to live value
12	sloss	Integer	Source packets retransmitted or dropped
13	dloss	Integer	Destination packets retransmitted or dropped
14	service	nominal	Service used
15	Sload	Float	Source bits per second
16	Dload	Float	Destination bits per second
17	Spkts	integer	Source to destination packet count
18	Dpkts	integer	Destination to source packet count
19	swin	integer	Source TCP window advertisement value
20	dwin	integer	Destination TCP window advertisement value
21	stcpb	integer	Source TCP base sequence number
22	dtcpb	integer	Destination TCP base sequence number
23	smeansz	integer	Mean of the flow packet size transmitted by the source
24	dmeansz	integer	Mean of the flow packet size transmitted by the destination
25	trans_depth	integer	Represents the pipelined depth into the connection of HTTP request/response transaction
26	res_bdy_len	integer	Actual uncompressed content size of the data transferred from the server’s HTTP service
27	Sjit	Float	Source jitter (milliseconds)
28	Djit	Float	Destination jitter (milliseconds)
29	Stime	Timestamp	Record start time
30	Ltime	Timestamp	Record last time
31	Sintpkt	Float	Source interpacket arrival time (milliseconds)
32	Dintpkt	Float	Destination interpacket arrival time (milliseconds)
33	tcprtt	Float	TCP connection setup round-trip time, the sum of ’synack’ and ’ackdat’
34	synack	Float	TCP connection setup time, the time between the SYN and the SYN_ACK packets
35	ackdat	Float	TCP connection setup time, the time between the SYN_ACK and the ACK packets
36	is_sm_ips_ports	Binary	If source and destination IP addresses equal and port numbers equal, this variable takes value 1
37	ct_state_ttl	Integer	No. for each state according to specific range of values for source/destination time to live
38	ct_flw_http_mthd	Integer	No. of flows that have methods such as Get and Post in HTTP service
39	is_ftp_login	Binary	If the FTP session is accessed by user and password then 1 else 0
40	ct_ftp_cmd	integer	No. of flows that have a command in FTP session
41	ct_srv_src	integer	No. of connections that contain the same service and source address in 100 connections
42	ct_srv_dst	integer	No. of connections that contain the same service and destination address in 100 connections
43	ct_dst_ltm	integer	No. of connections of the same destination address in 100 connections
44	ct_src_ltm	integer	No. of connections of the same source address in 100 connections
45	ct_src_dport_ltm	integer	No. of connections of the same source address and the destination port in 100 connections
46	ct_dst_sport_ltm	integer	No. of connections of the same destination address and the source port in 100 connections
47	ct_dst_src_ltm	integer	No. of connections of the same source and destination address in 100 connections
48	attack_cat	nominal	The name of each attack category
49	Label	binary	0 for normal and 1 for attack records

2. Big Data Query & Analysis by Apache Hive

This task involves utilizing Apache Hive to transform large raw data into actionable insights for end users. The process begins by thoroughly understanding the dataset. Subsequently, at least 4 Hive queries should be formulated (refer to the marking scheme). Suitable visualization tools should be applied to present the findings both numerically and graphically. A brief interpretation of the findings should also be provided.

Finally, screenshots of the outcomes, including tables and plots, along with the scripts/queries should be included in the report.

3. Advanced Analytics using PySpark 

In this section, you will conduct advanced analytics using PySpark.

3.1. Analyze and Interpret Big Data 

We need to learn and understand the data through at least 4 analytical methods (descriptive statistics, correlation, hypothesis testing, density estimation, etc.). You need to present your work numerically and graphically. Apply tooltip text, legend, title, X-Y labels etc. accordingly to help end-users for getting insights.

3.2. Design and Build a Classifier 

a) Design and build a binary classifier over the dataset. Explain your algorithm and its configuration. Explain your findings into both numerical and graphical representations. Evaluate the performance of the model and verify the accuracy and the effectiveness of your model. 

b) Apply a multi-class classifier to classify data into ten classes (categories): one normal and nine attacks (e.g., Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shellcode and Worms). Briefly explain your model with supportive statements on its parameters, accuracy and effectiveness.

3. Documentation 

Document all your work. Your final report must follow 5 sections detailed in the “format of final submission” section (refer to the next page). Your work must demonstrate appropriate understanding of academic writing and integrity.

Summarize the key points discussed in the blog post and reiterate the significance of the project in the context of big data analytics. Encourage readers to explore further and engage with the provided sample assignment.

---

Codersarts provides tailored assistance for your big data analytics project, following the outlined tasks and objectives in your blog. Our team specializes in guiding you through each stage of the project, from understanding the dataset to implementing advanced analytics techniques.

We offer hands-on support in utilizing Apache Hive and PySpark for data transformation, querying, and analysis. Our experts ensure efficient preprocessing and feature engineering to enhance the accuracy of your models. With a focus on coding best practices, we ensure the best and reliability of your analytical solutions.

Codersarts facilitates thorough project evaluation, conducting quantitative assessments and offering insightful interpretations of your findings. Additionally, we provide services such as documentation review and problem-solving sessions to enhance the overall quality and success of your big data analytics endeavor.

If you require any assistance with the project discussed in this blog, or if you find yourself in need of similar support for other projects, please don't hesitate to reach out to us. Our team can be contacted at any time via email at contact@codersarts.com.