db c2sr_testbed_aggregated

This database is structured into two primary collections, each serving a distinct purpose in our data aggregation system.

Collections

c2sr_anl_testbed_rl

Description:
This collection contains aggregated network data. It includes detailed records such as source and destination IP addresses, ports, protocols, and timestamps. The data here is compiled from various network flows and is used for analysis and monitoring purposes.
Data Dictionary for c2sr_anl_testbed_rl Collection

The following table provides a detailed data dictionary for a sample document from the c2sr_anl_testbed_rl collection. It explains what each field represents along with its data type.

Field Name	Description	Data Type
`_id`	Unique identifier automatically generated by MongoDB.	ObjectId
`netflow_src_ip`	Source IP address of the network flow.	String
`netflow_dst_ip`	Destination IP address of the network flow.	String
`netflow_src_port`	Source port number used in the network flow.	Number (Integer)
`netflow_dst_port`	Destination port number used in the network flow.	Number (Integer)
`netflow_protocol`	Protocol used (e.g., `tcp`, `udp`).	String
`netflow_tos`	Type of Service value from the IP header.	Number (Integer)
`netflow_flow_start`	Timestamp marking the start of the network flow (ISO 8601 format).	ISODate (String)
`netflow_flow_end`	Timestamp marking the end of the network flow (ISO 8601 format).	ISODate (String)
`netflow_flow_duration`	Duration of the network flow.	Number (Integer)
`netflow_packet_count`	Total number of packets observed in the flow.	Number (Integer)
`netflow_byte_count`	Total number of bytes transmitted during the flow.	Number (Integer)
`flow_input_interface_index`	Index of the input interface on the network device.	Number (Integer)
`netflow_output_interface_index`	Index of the output interface on the network device.	Number (Integer)
`netflow_tcp_flags`	TCP flags observed in the flow. (e.g., "20" represents specific flag settings.)	String
`netflow_src_asn`	Source Autonomous System Number (ASN), if applicable.	Number (Integer)
`netflow_dst_asn`	Destination Autonomous System Number (ASN), if applicable.	Number (Integer)
`netflow_next_hop`	Next hop IP address or identifier in the network path.	Number (Integer)
`netflow_icmp_type`	ICMP type (used if the flow involves ICMP).	Number (Integer)
`netflow_icmp_flags`	ICMP flags (used if the flow involves ICMP).	Number (Integer)
`netflow_icmp_code`	ICMP code (used if the flow involves ICMP).	Number (Integer)
`ids_misc-activity_src`	Miscellaneous IDS activity indicator or score related to the source.	Number (Integer)

Note: While the netflow_flow_start and netflow_flow_end fields are shown as strings in this example, they are typically stored as ISODate objects in MongoDB for proper date handling.

c2sr_testbed_serverless_stats

Description:
This collection holds performance statistics for our serverless functions. It tracks metrics such as query durations, document counts, processing times, and success rates. These statistics are essential for assessing the efficiency and reliability of our data aggregation processes.
Data Dictionary for c2sr_testbed_serverless_stats Collection

The following table provides a detailed data dictionary for a sample document from the c2sr_testbed_serverless_stats collection. It explains what each field represents along with its data type.

Field Name	Description	Data Type
`_id`	Unique identifier automatically generated by MongoDB.	ObjectId
`NetFlow Query docs count`	Count of documents returned by a NetFlow query.	Integer
`NetFlow Query duration`	Duration of the NetFlow query execution (in seconds).	Float
`n unique IPS`	Number of unique IP addresses encountered.	Integer
`NetFlow Processing duration`	Time taken to process NetFlow data (in seconds).	Float
`n ids docs`	Number of IDS documents processed.	Integer
`IDS query duration`	Duration of the IDS query execution (in seconds).	Float
`IDS Processing duration`	Time taken to process IDS data (in seconds).	Float
`n ADS Docs`	Number of ADS documents processed.	Integer
`ADS Query duration`	Duration of the ADS query execution (in seconds).	Float
`ADS Processing duration`	Time taken to process ADS data (in seconds).	Float
`Final Processing duration`	Total time for final processing of aggregated data (in seconds).	Float
`Final Bulk Insert Duration`	Duration of the bulk insert operation (in seconds).	Float
`final bulk insert`	Final count of documents inserted during the bulk insert.	Integer
`success`	Indicator of operation success (1 for success, 0 for failure).	Integer
`query_start_UTC`	Start time of the query in UTC (epoch time in milliseconds).	Long/Integer

c2sr_und_testbed

** Description**

This collection holds aggregated ids, ads and NEtflow data for the DOE project.

_id key

Field	Type	Description
`_id`	string	Unique MongoDB document identifier

netflow

Field	Type	Description
`netflow.src_ip`	string	Source IPv4 or IPv6 address of the flow
`netflow.dst_ip`	string	Destination IPv4 or IPv6 address of the flow
`netflow.src_port`	integer	Source transport-layer port number
`netflow.dst_port`	integer	Destination transport-layer port number
`netflow.protocol`	integer	IP protocol number (e.g., 6 = TCP, 17 = UDP)
`netflow.tos`	integer	IP “type of service” field value
`netflow.flow_start`	string	ISO-8601 timestamp when the flow began (UTC)
`netflow.flow_end`	string	ISO-8601 timestamp when the flow ended (UTC)
`netflow.flow_duration`	number	Duration of the flow in seconds (`flow_end – flow_start`)
`netflow.packet_count`	integer	Total number of packets in the flow
`netflow.byte_count`	integer	Total number of bytes in the flow
`netflow.flow_input_interface_index`	integer	Index of the ingress interface on which the flow was observed
`netflow.output_interface_index`	integer	Index of the egress interface on which the flow was forwarded
`netflow.tcp_flags`	integer	Bitmask representing the combined TCP flags seen in the flow
`netflow.src_asn`	integer or null	Autonomous System Number of the source (if resolved)
`netflow.dst_asn`	integer or null	Autonomous System Number of the destination (if resolved)
`netflow.next_hop`	string or null	IP address of the next hop for this flow
`netflow.icmp_type`	integer or null	ICMP type code (for ICMP flows)
`netflow.icmp_code`	integer or null	ICMP code (for ICMP flows)
`netflow.icmp_flags`	integer or null	ICMP flags (where supported)
`netflow.packet_variance`	number	Variance of packet sizes in the flow
`netflow.packet_mean`	number	Mean packet size in bytes
`netflow.packet_stddev`	number	Standard deviation of packet sizes
`netflow.packet_per_second`	number	Average packets per second
`netflow.byte_per_second`	number	Average bytes per second
`netflow.max_packet_length`	integer	Maximum observed packet length
`netflow.min_packet_length`	integer	Minimum observed packet length
`netflow.tcp_F_count`	integer	Number of TCP FIN flags seen
`netflow.tcp_S_count`	integer	Number of TCP SYN flags seen
`netflow.tcp_R_count`	integer	Number of TCP RST flags seen
`netflow.tcp_P_count`	integer	Number of TCP PSH flags seen
`netflow.tcp_A_count`	integer	Number of TCP ACK flags seen
`netflow.tcp_U_count`	integer	Number of TCP URG flags seen
`netflow.tcp_E_count`	integer	Number of TCP ECE flags seen
`netflow.tcp_C_count`	integer	Number of TCP CWR flags seen

IDS

Field	Type	Description
`ids.src_ip_alerts.attempted-admin`	integer	Count of “attempted-admin” alerts
`ids.src_ip_alerts.attempted-dos`	integer	Count of “attempted-dos” alerts
`ids.src_ip_alerts.attempted-recon`	integer	Count of “attempted-recon” alerts
`ids.src_ip_alerts.attempted-user`	integer	Count of “attempted-user” alerts
`ids.src_ip_alerts.bad-unknown`	integer	Count of “bad-unknown” alerts
`ids.src_ip_alerts.coin-mining`	integer	Count of “coin-mining” alerts
`ids.src_ip_alerts.command-and-control`	integer	Count of “command-and-control” alerts
`ids.src_ip_alerts.credential-theft`	integer	Count of “credential-theft” alerts
`ids.src_ip_alerts.default-login-attempt`	integer	Count of “default-login-attempt” alerts
`ids.src_ip_alerts.denial-of-service`	integer	Count of “denial-of-service” alerts
`ids.src_ip_alerts.domain-c2`	integer	Count of “domain-c2” alerts
`ids.src_ip_alerts.exploit-kit`	integer	Count of “exploit-kit” alerts
`ids.src_ip_alerts.external-ip-check`	integer	Count of “external-ip-check” alerts
`ids.src_ip_alerts.icmp-event`	integer	Count of “icmp-event” alerts
`ids.src_ip_alerts.misc-activity`	integer	Count of “misc-activity” alerts
`ids.src_ip_alerts.misc-attack`	integer	Count of “misc-attack” alerts
`ids.src_ip_alerts.network-scan`	integer	Count of “network-scan” alerts
`ids.src_ip_alerts.non-standard-protocol`	integer	Count of “non-standard-protocol” alerts
`ids.src_ip_alerts.not-suspicious`	integer	Count of “not-suspicious” alerts
`ids.src_ip_alerts.policy-violation`	integer	Count of “policy-violation” alerts
`ids.src_ip_alerts.protocol-command-decode`	integer	Count of “protocol-command-decode” alerts
`ids.src_ip_alerts.pup-activity`	integer	Count of “pup-activity” alerts
`ids.src_ip_alerts.rpc-portmap-decode`	integer	Count of “rpc-portmap-decode” alerts
`ids.src_ip_alerts.shellcode-detect`	integer	Count of “shellcode-detect” alerts
`ids.src_ip_alerts.social-engineering`	integer	Count of “social-engineering” alerts
`ids.src_ip_alerts.string-detect`	integer	Count of “string-detect” alerts
`ids.src_ip_alerts.successful-admin`	integer	Count of “successful-admin” alerts
`ids.src_ip_alerts.successful-dos`	integer	Count of “successful-dos” alerts
`ids.src_ip_alerts.successful-recon-largescale`	integer	Count of “successful-recon-largescale” alerts
`ids.src_ip_alerts.successful-recon-limited`	integer	Count of “successful-recon-limited” alerts
`ids.src_ip_alerts.successful-user`	integer	Count of “successful-user” alerts
`ids.src_ip_alerts.suspicious-filename-detect`	integer	Count of “suspicious-filename-detect” alerts
`ids.src_ip_alerts.suspicious-login`	integer	Count of “suspicious-login” alerts
`ids.src_ip_alerts.system-call-detect`	integer	Count of “system-call-detect” alerts
`ids.src_ip_alerts.targeted-activity`	integer	Count of “targeted-activity” alerts
`ids.src_ip_alerts.tcp-connection`	integer	Count of “tcp-connection” alerts
`ids.src_ip_alerts.trojan-activity`	integer	Count of “trojan-activity” alerts
`ids.src_ip_alerts.unknown`	integer	Count of “unknown” alerts
`ids.src_ip_alerts.unsuccessful-user`	integer	Count of “unsuccessful-user” alerts
`ids.src_ip_alerts.unusual-client-port-connection`	integer	Count of “unusual-client-port-connection” alerts
`ids.src_ip_alerts.web-application-activity`	integer	Count of “web-application-activity” alerts
`ids.src_ip_alerts.web-application-attack`	integer	Count of “web-application-attack” alerts

Field	Type	Description
`ads.anomaly_score`	number	Numeric score from the anomaly detection system (–1 if no valid score, -2 = ADS failed to produce response)
`ads.anomaly_flag`	integer	Flag from the ADS (–1 = error or no data; 0 = normal; 1 = anomalous flow; -2 = ADS failed to produce response)

c2sr_testbed_und_serverless_stats

Data Dictionary for c2sr_testbed_serverless_stats Collection