Kubernetes Testbed: Checking and Launching Microservices

Overview

This guide explains how to check and start up services deployed as microservices within Kubernetes pods on the testbed. This is particularly useful if you are looking to replicate our testbed setup.

Before you start

Before you check or start up services, ensure that:

All the services are deployed in the Kubernetes cluster.
All the services are running correctly.

Services, what is does and how to access it.

Some service names are stripped of the c2sr-testbed- prefix to save space. Adding this prefix will match the Github repo names.

data-hub

data-hub is a basic ui that provides a basic user interface for the user to select which csv file streams for which inverter emulator, when the user makes the selection, the file name is sent over to data-weaver over the device's kafka topic.

This service runs on the group:prem-cloud as a daemaonset. It is a node port service that listens on port 5000 for traffic which is exposted to the node port on port 32500, and allows external traffic (from outside the cluster). To access the UI openthe following snippet in a browser from a computer on the same network as the device hosting the service.

<node-ip>:32500

ansible@ip-172-31-25-157:~$ k logs -f stream-selecter-zm2sk
status_updater started:  True
Topic nano01_stream_ack created successfully
Topic nano02_stream_ack created successfully
Topic nano03_stream_ack created successfully
Topic nano04_stream_ack created successfully
Topic nano05_stream_ack created successfully
Topic nano06_stream_ack created successfully
Topic redteam1_stream_ack created successfully
Consumer created successfully!
Consumer joined ['nano01_stream_ack', 'nano02_stream_ack', 'nano03_stream_ack', 'nano04_stream_ack', 'nano05_stream_ack', 'nano06_stream_ack', 'redteam1_stream_ack'].
Dash is running on http://0.0.0.0:5000/

 * Serving Flask app 'main'
 * Debug mode: on
%0|1740286144.571|FATAL|rdkafka#consumer-9| [thrd:main]: Fatal error: Broker: Static consumer fenced by other consumer with same group.instance.id: Fatal consumer error: Broker: Static consumer fenced by other consumer with same group.instance.id

data-weaver

The data weaver listens for the CSV file name transferred form the UI, downloads the file from S3, and streams its data over the device's kafka topic.

The service does not expose any UIs, inspection of logs would be sufficient to monitor the execution. In addition to the logs, as the download, and switch of CSV happens, status updates are posed via Kafka to the UI.

ansible@ip-172-31-25-157:~$ k logs -f inverter-streamer-xc44f
2025-02-23 04:48:38,858 - __main__ - INFO - The inverter Streamer will generate data for: ['nano01', 'nano02', 'nano03', 'nano04', 'nano05', 'nano06']
2025-02-23 04:48:38,867 - __main__ - INFO - Starting topic creation: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
2025-02-23 04:48:38,867 - __main__ - INFO - Creating topic: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
%4|1740286118.993|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire idempotence PID from broker kafka-cloud-service:9092/1001: Broker: Coordinator load in progress: retrying
2025-02-23 04:48:39,209 - kafka.kafka_admin - INFO - Topic nano01_stream_file created successfully
2025-02-23 04:48:39,210 - __main__ - INFO - Creating topic: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
2025-02-23 04:48:39,354 - kafka.kafka_admin - INFO - Topic nano02_stream_file created successfully
2025-02-23 04:48:39,355 - __main__ - INFO - Creating topic: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
2025-02-23 04:48:39,494 - kafka.kafka_admin - INFO - Topic nano03_stream_file created successfully
2025-02-23 04:48:39,495 - __main__ - INFO - Creating topic: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
2025-02-23 04:48:39,638 - kafka.kafka_admin - INFO - Topic nano04_stream_file created successfully
2025-02-23 04:48:39,639 - __main__ - INFO - Creating topic: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
2025-02-23 04:48:39,789 - kafka.kafka_admin - INFO - Topic nano05_stream_file created successfully
2025-02-23 04:48:39,789 - __main__ - INFO - Creating topic: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
2025-02-23 04:48:39,921 - kafka.kafka_admin - INFO - Topic nano06_stream_file created successfully
2025-02-23 04:48:39,922 - __main__ - INFO - Creating topic: ['nano01_stream_file', 'nano02_stream_file', 'nano03_stream_file', 'nano04_stream_file', 'nano05_stream_file', 'nano06_stream_file', 'redteam1_stream_file']
2025-02-23 04:48:40,064 - kafka.kafka_admin - INFO - Topic redteam1_stream_file created successfully
2025-02-23 04:48:41,066 - __main__ - INFO - =============================================
.
.
.
.
2025-02-23 04:51:23,117 - __main__ - INFO - =============================================
2025-02-23 04:51:24,118 - __main__ - INFO - Received update: nano01: 494654_SF
2025-02-23 04:51:24,118 - thread_manager - INFO - nano01 is streaming: Awaiting Start
2025-02-23 04:51:24,118 - thread_manager - INFO - nano01 will be switched to: 494654_SF
2025-02-23 04:51:24,118 - thread_manager - INFO - nano01: Reading file: 494654_SF.
2025-02-23 04:51:38,425 - thread_manager - INFO - nano01: 494654_SF Read.
2025-02-23 04:51:38,425 - thread_manager - INFO - Starting Fresh stream..
2025-02-23 04:51:38,429 - thread_manager - INFO - nano01 thread: 494654_SF started.
2025-02-23 04:51:38,429 - __main__ - INFO - =============================================
. 
. 
.
2025-02-23 04:52:52,455 - __main__ - INFO - =============================================
2025-02-23 04:52:53,455 - __main__ - INFO - Received update: nano02: 494654_SF
2025-02-23 04:52:53,455 - thread_manager - INFO - nano02 is streaming: Awaiting Start
2025-02-23 04:52:53,456 - thread_manager - INFO - nano02 will be switched to: 494654_SF
2025-02-23 04:52:53,456 - thread_manager - INFO - nano02: Reading file: 494654_SF.
2025-02-23 04:53:05,173 - thread_manager - INFO - nano02: 494654_SF Read.
2025-02-23 04:53:05,173 - thread_manager - INFO - Starting Fresh stream..
2025-02-23 04:53:05,174 - thread_manager - INFO - nano02 thread: 494654_SF started.
2025-02-23 04:53:05,174 - __main__ - INFO - =============================================
2025-02-23 04:53:39,323 - __main__ - INFO - =============================================
2025-02-23 04:53:41,323 - __main__ - INFO - =============================================
2025-02-23 04:53:43,324 - __main__ - INFO - =============================================
2025-02-23 04:53:45,325 - __main__ - INFO - =============================================
2025-02-23 04:53:47,325 - __main__ - INFO - =============================================

suns-inv

This service consumes from the device specific kafka topic, updates the SunSpec inverter emulation with new data produced from the data-weaver.

ansible@ip-172-31-25-157:~$ k logs inverter-9hplc
2025-02-23 04:49:14,193 - __main__ - INFO - Logging level: INFO
2025-02-23 04:49:14,193 - __main__ - INFO - NODE_IP: 100.114.246.114
2025-02-23 04:49:14,193 - __main__ - INFO - NODE_NAME: nano02
2025-02-23 04:49:14,197 - kafka.consumer - INFO - Subscribed to input_nano02
2025-02-23 04:49:14,197 - __main__ - INFO - Device name is: nano02
2025-02-23 04:49:14,197 - __main__ - INFO - Adjusted device_name: nano02
2025-02-23 04:49:14,197 - __main__ - INFO - Manufacturer: c2sr
2025-02-23 04:49:14,197 - __main__ - INFO - Model: nano02
2025-02-23 04:49:14,198 - __main__ - INFO - Opts: GenOpts
2025-02-23 04:49:14,198 - __main__ - INFO - Version: GenVer
2025-02-23 04:49:14,198 - __main__ - INFO - Serial No: GenericSlNo
2025-02-23 04:49:14,198 - __main__ - INFO - Address: 114
2025-02-23 04:49:14,198 - __main__ - INFO - Adding myself to active devices:
2025-02-23 04:49:14,198 - server_main - INFO - Starting server
2025-02-23 04:49:14,199 - server_main - INFO - Server is online: Host: 0.0.0.0, Port: 502
2025-02-23 04:49:14,375 - kafka.consumer - INFO - Awaiting creation of topic:  input_nano02
2025-02-23 04:53:19,134 - server_main - INFO - Txed: 10th msg: {'AphA': 0, 'A_SF': -3, 'Conn': 1.0, 'Conn_WinTms': 0.0, 'Hz': 0, 'Hz_SF': -3, 'OutPFSet': 10.0, 'OutPFSet_RmpTms': 0.0, 'PF': 10000, 'PF_SF': -2, 'PhVphA': 0, 'V_SF': -2, 'Ris': 655.35, 'St': 7.0, 'StActCtl': 0.0, 'VA': 0, 'VA_SF': 0, 'VAMax': 563.25, 'VAr': 0, 'VAr_SF': -3, 'VArMaxPct': 0.0, 'VArPct_RmpTms': 0.0, 'VRef': 614.4, 'W': 0, 'W_SF': 0, 'WH': 826660720, 'WH_SF': -1, 'WMaxLimPct': 100.0, 'WMaxLimPct_RmpTms': 0.0, 'DCV': 280, 'DCV_SF': -2, 'DCA': 0, 'DCA_SF': -2, 'DCW': 0, 'DCW_SF': -1}
2025-02-23 04:53:29,311 - server_main - INFO - Txed: 20th msg: {'AphA': 0, 'A_SF': -3, 'Conn': 1.0, 'Conn_WinTms': 0.0, 'Hz': 0, 'Hz_SF': -3, 'OutPFSet': 10.0, 'OutPFSet_RmpTms': 0.0, 'PF': 10000, 'PF_SF': -2, 'PhVphA': 0, 'V_SF': -2, 'Ris': 655.35, 'St': 7.0, 'StActCtl': 0.0, 'VA': 0, 'VA_SF': 0, 'VAMax': 563.25, 'VAr': 0, 'VAr_SF': -3, 'VArMaxPct': 0.0, 'VArPct_RmpTms': 0.0, 'VRef': 614.4, 'W': 0, 'W_SF': 0, 'WH': 826660720, 'WH_SF': -1, 'WMaxLimPct': 100.0, 'WMaxLimPct_RmpTms': 0.0, 'DCV': 280, 'DCV_SF': -2, 'DCA': 0, 'DCA_SF': -2, 'DCW': 0, 'DCW_SF': -1}
2025-02-23 04:53:39,492 - server_main - INFO - Txed: 30th msg: {'AphA': 0, 'A_SF': -3, 'Conn': 1.0, 'Conn_WinTms': 0.0, 'Hz': 0, 'Hz_SF': -3, 'OutPFSet': 10.0, 'OutPFSet_RmpTms': 0.0, 'PF': 10000, 'PF_SF': -2, 'PhVphA': 0, 'V_SF': -2, 'Ris': 655.35, 'St': 7.0, 'StActCtl': 0.0, 'VA': 0, 'VA_SF': 0, 'VAMax': 563.25, 'VAr': 0, 'VAr_SF': -3, 'VArMaxPct': 0.0, 'VArPct_RmpTms': 0.0, 'VRef': 614.4, 'W': 0, 'W_SF': 0, 'WH': 826660720, 'WH_SF': -1, 'WMaxLimPct': 100.0, 'WMaxLimPct_RmpTms': 0.0, 'DCV': 280, 'DCV_SF': -2, 'DCA': 0, 'DCA_SF': -2, 'DCW': 0, 'DCW_SF': -1}
2025-02-23 04:53:51,071 - server_main - INFO - Txed: 40th msg: {'AphA': 0, 'A_SF': -3, 'Conn': 1.0, 'Conn_WinTms': 0.0, 'Hz': 0, 'Hz_SF': -3, 'OutPFSet': 10.0, 'OutPFSet_RmpTms': 0.0, 'PF': 10000, 'PF_SF': -2, 'PhVphA': 0, 'V_SF': -2, 'Ris': 655.35, 'St': 7.0, 'StActCtl': 0.0, 'VA': 0, 'VA_SF': 0, 'VAMax': 563.25, 'VAr': 0, 'VAr_SF': -3, 'VArMaxPct': 0.0, 'VArPct_RmpTms': 0.0, 'VRef': 614.4, 'W': 0, 'W_SF': 0, 'WH': 826660720, 'WH_SF': -1, 'WMaxLimPct': 100.0, 'WMaxLimPct_RmpTms': 0.0, 'DCV': 32768, 'DCV_SF': -2, 'DCA': 65535, 'DCA_SF': -2, 'DCW': 2147418, 'DCW_SF': -1}

suns-seek

This service queries the SunSpec Modbus TCP inverter to get the inverter data, and creates SunSpec Modubs traffic on the network. Once it obtains the data, it writes it to a kafka topic.

Logs of this microservice are self-explanatory and can be used to know if it is running as expected. Look for entries like; nano0x Rx. Rx is shortened form of Received. To show suns-seek was able to query the specific inverter.

2025-02-23 05:27:20,283 - __main__ - INFO - Logging level: INFO
2025-02-23 05:27:20,283 - __main__ - INFO - My ip is: 100.82.142.60
2025-02-23 05:27:20,283 - __main__ - INFO - My node name is: agx02
2025-02-23 05:27:20,283 - __main__ - INFO - My device name is: agx02
2025-02-23 05:27:20,283 - __main__ - INFO - I am: agx02.
2025-02-23 05:27:20,283 - __main__ - INFO - I will connect to: ['nano03', 'nano04'].
2025-02-23 05:27:20,283 - __main__ - INFO - Attempting Connections...
2025-02-23 05:27:20,286 - __main__ - INFO - Starting to monitor: nano03 at 192.168.1.13, 30502
2025-02-23 05:27:20,286 - __main__ - INFO - Starting to monitor: nano04 at 192.168.1.14, 30502
2025-02-23 05:27:20,299 - client_main - INFO - nano03: Rx.
2025-02-23 05:27:20,300 - client_main - INFO - nano04: Rx.
2025-02-23 05:27:22,306 - client_main - INFO - nano04: Rx.
2025-02-23 05:27:22,314 - client_main - INFO - nano03: Rx.

data-batch-aggregator

This service consumes from the kafka topics that suns-seek writes to, aggregates the data into different time intervals, imputes missing data, using the forward fill, and moving averages, and produces to a kafka topic.

Logs of this microservice are self-explanatory and can be used to know if it is running as expected.

2025-02-23 05:27:15,376 - __main__ - INFO - Logging level: INFO
2025-02-23 05:27:15,377 - __main__ - INFO - I am: agx01
2025-02-23 05:27:15,377 - __main__ - INFO - My device name is: agx01
2025-02-23 05:27:15,377 - __main__ - INFO - I will connect to : ['nano01', 'nano02']
2025-02-23 05:27:15,377 - __main__ - INFO - Creating Consumers:
2025-02-23 05:27:15,377 - __main__ - INFO - Creating consumer: nano01
2025-02-23 05:27:15,384 - __main__ - INFO - Creating consumer: nano02
2025-02-23 05:27:15,384 - __main__ - INFO - Consumers created.
2025-02-23 05:27:15,385 - __main__ - INFO - Creating producer.
2025-02-23 05:27:15,385 - __main__ - INFO - Producer created.
2025-02-23 05:27:15,479 - root - INFO - All produced topics: ['nano02_inverter_raw', 'nano01_inverter_raw']
2025-02-23 05:27:15,479 - root - INFO - Topics to query are: ['nano02_inverter_raw', 'nano01_inverter_raw']
2025-02-23 05:27:15,479 - __main__ - INFO - 0 queried threads: []
2025-02-23 05:27:15,479 - __main__ - INFO - Found 2 un-queried topics: ['nano01_inverter_raw', 'nano02_inverter_raw']
2025-02-23 05:27:15,479 - __main__ - INFO - Starting to query nano01_inverter_raw
2025-02-23 05:27:15,483 - __main__ - INFO - Starting to query nano02_inverter_raw
2025-02-23 05:27:15,575 - root - INFO - All produced topics: ['nano02_inverter_raw', 'nano01_inverter_raw']
2025-02-23 05:27:15,575 - root - INFO - Topics to query are: ['nano02_inverter_raw', 'nano01_inverter_raw']
2025-02-23 05:27:15,575 - __main__ - INFO - 2 queried threads: ['nano01_inverter_raw', 'nano02_inverter_raw']
2025-02-23 05:27:15,575 - __main__ - INFO - Found 0 un-queried topics: []
2025-02-23 05:27:18,709 - root - INFO - Collecting Fwd fill imputation values for nano01
2025-02-23 05:27:18,754 - root - INFO - Collecting Moving Avg imputation values for nano01
2025-02-23 05:27:19,712 - root - INFO - Transmitting fwd_fill data for nano01
2025-02-23 05:27:19,735 - root - INFO - Imputing Fwd fill.
2025-02-23 05:27:19,735 - root - INFO - Collecting Fwd fill imputation values for nano02
2025-02-23 05:27:19,758 - root - INFO - Collecting Moving Avg imputation values for nano02
2025-02-23 05:27:20,671 - root - INFO - All produced topics: ['nano02_inverter_raw', 'nano03_inverter_raw', 'nano04_inverter_raw', 'nano01_inverter_raw']
2025-02-23 05:27:20,671 - root - INFO - Topics to query are: ['nano02_inverter_raw', 'nano01_inverter_raw']
2025-02-23 05:27:20,671 - __main__ - INFO - 2 queried threads: ['nano01_inverter_raw', 'nano02_inverter_raw']
2025-02-23 05:27:20,671 - __main__ - INFO - Found 0 un-queried topics: []
.
.
.
2025-02-23 06:17:11,610 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:11,627 - root - INFO - Imputing with moving mean: 5
2025-02-23 06:17:11,662 - root - INFO - Transmitting fwd_fill data for nano02
2025-02-23 06:17:11,687 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:11,712 - root - INFO - Continuing without imputation.
2025-02-23 06:17:12,614 - root - INFO - Transmitting fwd_fill data for nano01
2025-02-23 06:17:12,623 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:12,643 - root - INFO - Continuing without imputation.
2025-02-23 06:17:12,678 - root - INFO - Transmitting fwd_fill data for nano02
2025-02-23 06:17:12,687 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:12,708 - root - INFO - Continuing without imputation.
2025-02-23 06:17:13,629 - root - INFO - Transmitting fwd_fill data for nano01
2025-02-23 06:17:13,639 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:13,659 - root - INFO - Continuing without imputation.
2025-02-23 06:17:13,694 - root - INFO - Transmitting fwd_fill data for nano02
2025-02-23 06:17:13,706 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:13,726 - root - INFO - Continuing without imputation.
2025-02-23 06:17:14,644 - root - INFO - Transmitting fwd_fill data for nano01
2025-02-23 06:17:14,654 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:14,676 - root - INFO - Continuing without imputation.
2025-02-23 06:17:14,711 - root - INFO - Transmitting fwd_fill data for nano02
2025-02-23 06:17:14,720 - root - INFO - Imputing Fwd fill.
2025-02-23 06:17:14,740 - root - INFO - Continuing without imputation.

data-windower

This service consumes from the kafka topics that data-batch-aggregator writes to, windows the data into 128 sized windows, and calls the ADS, receives the response, writes it to OpenSearch, and AWS Firehose using boto3.

Logs of this microservice are self-explanatory and can be used to know if it is running as expected.

ansible@ip-172-31-25-157:~$ k logs -f ads-caller-426jt
2025-02-23 06:10:04,125 - __main__ - INFO - MainThread - Logging level: DEBUG
2025-02-23 06:10:04,126 - __main__ - INFO - MainThread - I am: agx02
2025-02-23 06:10:04,126 - __main__ - INFO - MainThread - My device name is: agx02
2025-02-23 06:10:04,126 - __main__ - INFO - MainThread - I will connect to : ['nano03', 'nano04']
2025-02-23 06:10:04,126 - __main__ - INFO - MainThread - Creating Consumers:
2025-02-23 06:10:04,126 - __main__ - INFO - MainThread - Creating consumer: nano03
2025-02-23 06:10:04,128 - __main__ - INFO - MainThread - Creating consumer: nano04
2025-02-23 06:10:04,129 - __main__ - INFO - MainThread - Consumers created.
2025-02-23 06:10:04,212 - root - INFO - MainThread - Currently active topics are: ['nano01_t1s', 'nano02_t1s', 'nano03_t5m', 'input_nano03', 'nano04_fwd_filled', 'input_nano04', 'redteam1_stream_file', 'nano06_stream_file', 'nano03_stream_ack', 'redteam1_stream_ack', 'nano02_stream_ack', 'nano04_t1m', 'nano02_inverter_raw', 'nano05_stream_ack', 'input_nano02', 'nano03_t1m', 'nano03_inverter_raw', 'nano02_fwd_filled', 'nano03_fwd_filled', '__consumer_offsets', 'nano02_stream_file', 'nano01_stream_ack', 'nano05_stream_file', 'nano01_t5m', 'nano01_stream_file', 'nano04_inverter_raw', 'nano02_t5m', 'nano01_fwd_filled', 'nano01_inverter_raw', 'input_nano01', 'nano04_t5m', 'nano03_stream_file', 'nano06_stream_ack', 'nano02_t1m', 'nano04_t1s', 'nano01_t1m', 'nano03_t1s', 'nano04_stream_ack', 'nano04_stream_file']
2025-02-23 06:10:04,212 - root - INFO - MainThread - Currently active topics are: ['nano04_fwd_filled', 'nano02_fwd_filled', 'nano03_fwd_filled', 'nano01_fwd_filled']
2025-02-23 06:10:04,212 - root - INFO - MainThread - The topics to query are: ['nano04_fwd_filled', 'nano03_fwd_filled']
2025-02-23 06:10:04,212 - __main__ - INFO - MainThread - 0 queried threads: []
2025-02-23 06:10:04,212 - __main__ - INFO - MainThread - Found 2 un-queried topics: ['nano03_fwd_filled', 'nano04_fwd_filled']
2025-02-23 06:10:04,213 - ads_caller - INFO - nano03_fwd_filled - nano03_fwd_filled
2025-02-23 06:10:04,213 - ads_caller - INFO - nano03_fwd_filled - Function Triggered
2025-02-23 06:10:04,213 - root - INFO - nano03_fwd_filled - Collecting messages from nano03_fwd_filled
2025-02-23 06:10:04,213 - ads_caller - INFO - nano04_fwd_filled - nano04_fwd_filled
2025-02-23 06:10:04,214 - ads_caller - INFO - nano04_fwd_filled - Function Triggered
2025-02-23 06:10:04,214 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:04,290 - root - INFO - MainThread - Currently active topics are: ['nano01_t1s', 'nano02_t1s', 'nano03_t5m', 'input_nano03', 'nano04_fwd_filled', 'input_nano04', 'redteam1_stream_file', 'nano06_stream_file', 'nano03_stream_ack', 'redteam1_stream_ack', 'nano02_stream_ack', 'nano04_t1m', 'nano02_inverter_raw', 'nano05_stream_ack', 'input_nano02', 'nano03_t1m', 'nano03_inverter_raw', 'nano02_fwd_filled', 'nano03_fwd_filled', '__consumer_offsets', 'nano02_stream_file', 'nano01_stream_ack', 'nano05_stream_file', 'nano01_t5m', 'nano01_stream_file', 'nano04_inverter_raw', 'nano02_t5m', 'nano01_fwd_filled', 'nano01_inverter_raw', 'input_nano01', 'nano04_t5m', 'nano03_stream_file', 'nano06_stream_ack', 'nano02_t1m', 'nano04_t1s', 'nano01_t1m', 'nano03_t1s', 'nano04_stream_ack', 'nano04_stream_file']
2025-02-23 06:10:04,291 - root - INFO - MainThread - Currently active topics are: ['nano04_fwd_filled', 'nano02_fwd_filled', 'nano03_fwd_filled', 'nano01_fwd_filled']
2025-02-23 06:10:04,291 - root - INFO - MainThread - The topics to query are: ['nano04_fwd_filled', 'nano03_fwd_filled']
2025-02-23 06:10:04,291 - __main__ - INFO - MainThread - 2 queried threads: ['nano03_fwd_filled', 'nano04_fwd_filled']
2025-02-23 06:10:04,291 - __main__ - INFO - MainThread - Found 0 un-queried topics: []
2025-02-23 06:10:04,291 - __main__ - INFO - MainThread - All topics are queried.
2025-02-23 06:10:04,291 - __main__ - DEBUG - MainThread - Queried topics are ['MainThread', 'nano03_fwd_filled', 'nano04_fwd_filled']
2025-02-23 06:10:04,372 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 1 Awaiting 127 more messages.
2025-02-23 06:10:04,372 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:04,877 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 2 Awaiting 126 more messages.
2025-02-23 06:10:04,877 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:04,937 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 3 Awaiting 125 more messages.
2025-02-23 06:10:04,937 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:05,446 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 4 Awaiting 124 more messages.
2025-02-23 06:10:05,446 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:05,936 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 5 Awaiting 123 more messages.
2025-02-23 06:10:05,936 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:06,444 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 6 Awaiting 122 more messages.
2025-02-23 06:10:06,444 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:06,956 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 7 Awaiting 121 more messages.
2025-02-23 06:10:06,956 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:06,980 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 8 Awaiting 120 more messages.
2025-02-23 06:10:06,980 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:10:07,300 - ads_caller - INFO - nano03_fwd_filled - Awaiting window size (128). Current size: 1 Awaiting 127 more messages.
2025-02-23 06:10:07,300 - root - INFO - nano03_fwd_filled - Collecting messages from nano03_fwd_filled
.
.
.
2025-02-23 06:12:39,788 - ads_caller - INFO - nano04_fwd_filled - Awaiting window size (128). Current size: 127 Awaiting 1 more messages.
2025-02-23 06:12:39,788 - root - INFO - nano04_fwd_filled - Collecting messages from nano04_fwd_filled
2025-02-23 06:12:40,293 - ads_caller - INFO - nano04_fwd_filled - Calling ADS.
2025-02-23 06:12:41,155 - ads_caller - INFO - nano03_fwd_filled - Response From ADS: {'anomaly_flag': True, 'anomaly_score': 1.0}
2025-02-23 06:12:41,156 - ads_caller - INFO - nano03_fwd_filled - Detection: 1
2025-02-23 06:12:41,156 - ads_caller - INFO - nano03_fwd_filled - Writing data to elasticsearch
/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lambda-scalar.ad.und.edu'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2025-02-23 06:12:41,545 - elastic_transport.transport - INFO - nano03_fwd_filled - POST https://lambda-scalar.ad.und.edu:9200/c2sr-testbed-ads-2025.02.23/_doc [status:201 duration:0.330s]
2025-02-23 06:12:41,546 - ads_caller - INFO - nano03_fwd_filled - Elastic Search Resp: {'_index': 'c2sr-testbed-ads-2025.02.23', '_id': 'sStvMZUB-8HEccerveLD', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 6, '_primary_term': 1}
2025-02-23 06:12:41,546 - ads_caller - INFO - nano03_fwd_filled - Writing data to OpenSearch
2025-02-23 06:12:41,882 - root - INFO - MainThread - Currently active topics are: ['nano01_t1s', 'nano02_t1s', 'nano03_t5m', 'input_nano03', 'nano04_fwd_filled', 'input_nano04', 'redteam1_stream_file', 'nano06_stream_file', 'nano03_stream_ack', 'redteam1_stream_ack', 'nano02_stream_ack', 'nano04_t1m', 'nano02_inverter_raw', 'nano05_stream_ack', 'input_nano02', 'nano03_t1m', 'nano03_inverter_raw', 'nano02_fwd_filled', 'nano03_fwd_filled', '__consumer_offsets', 'nano02_stream_file', 'nano01_stream_ack', 'nano05_stream_file', 'nano01_t5m', 'nano01_stream_file', 'nano04_inverter_raw', 'nano02_t5m', 'nano01_fwd_filled', 'nano01_inverter_raw', 'input_nano01', 'nano04_t5m', 'nano03_stream_file', 'nano06_stream_ack', 'nano02_t1m', 'nano04_t1s', 'nano01_t1m', 'nano03_t1s', 'nano04_stream_ack', 'nano04_stream_file']
2025-02-23 06:12:41,882 - root - INFO - MainThread - Currently active topics are: ['nano04_fwd_filled', 'nano02_fwd_filled', 'nano03_fwd_filled', 'nano01_fwd_filled']
2025-02-23 06:12:41,883 - root - INFO - MainThread - The topics to query are: ['nano04_fwd_filled', 'nano03_fwd_filled']
2025-02-23 06:12:41,883 - __main__ - INFO - MainThread - 2 queried threads: ['nano03_fwd_filled', 'nano04_fwd_filled']
2025-02-23 06:12:41,883 - __main__ - INFO - MainThread - Found 0 un-queried topics: []
2025-02-23 06:12:41,883 - __main__ - INFO - MainThread - All topics are queried.
2025-02-23 06:12:41,883 - __main__ - DEBUG - MainThread - Queried topics are ['MainThread', 'nano03_fwd_filled', 'nano04_fwd_filled']
/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lambda-scalar.ad.und.edu'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2025-02-23 06:12:41,907 - opensearch - INFO - nano03_fwd_filled - POST https://lambda-scalar.ad.und.edu:9201/c2sr-testbed-ads-2025.02.23/_doc [status:201 request:0.360s]
2025-02-23 06:12:41,907 - opensearch - DEBUG - nano03_fwd_filled - > {"timestamp_ms":1740291161156,"timestamp_ns":1740291161156155484,"anomaly_score":1.0,"anomaly_flag":1,"inverter_ip":"192.168.1.13","edge_device_ip":"192.168.1.22"}
2025-02-23 06:12:41,907 - opensearch - DEBUG - nano03_fwd_filled - < {"_index":"c2sr-testbed-ads-2025.02.23","_id":"uURvMZUBTrvr3l-uvxAt","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":6,"_primary_term":1}
2025-02-23 06:12:41,908 - ads_caller - INFO - nano03_fwd_filled - OpenSearch Resp: {'_index': 'c2sr-testbed-ads-2025.02.23', '_id': 'uURvMZUBTrvr3l-uvxAt', '_version': 1, 'result': 'created', '_shards': {'total': 1, 'successful': 1, 'failed': 0}, '_seq_no': 6, '_primary_term': 1}
2025-02-23 06:12:41,908 - ads_caller - INFO - nano03_fwd_filled - Success

Last modified: 23 February 2025