Overview
Service health overview
2 hosts ·
48 ok · 50 failing · 0 unknown
auto-refresh every 20s · generated 2026-07-14T10:30:28+00:00
▸ test ssh-poller
monitoring sp_monitor internal — the monitor watching itself 1 failing 0 ok
| Service | Check | Status | Message | Last seen |
|---|---|---|---|---|
_agent_ |
agent_watchdog | fail | SSH connection failed: No authentication methods available | 12s ago |
▸ thor ssh-poller
cpu
34.8%
load 25.42 / 22.96 / 23.54 on 128 cores
ram
23.6%
115.0 GB used of 503.5 GB
disk
93.2%
58.6 GB free of 915.3 GB
iris IRIS pipeline + identifier + supplier 4 failing 4 ok
| Service | Check | Status | Message | Last seen |
|---|---|---|---|---|
iris_v3_captioning:0 |
supervisor | fail | iris_v3_captioning:0 STOPPED — Not started | 4m ago |
iris_v3_captioning:1 |
supervisor | fail | iris_v3_captioning:1 STOPPED — Not started | 4m ago |
iris_v3_captioning:2 |
supervisor | fail | iris_v3_captioning:2 STOPPED — Not started | 4m ago |
iris_v3_categorization:0 |
supervisor | fail | iris_v3_categorization:0 STOPPED — Not started | 4m ago |
iris-v3-gpu-worker-anchor |
docker | ok | container running | 4m ago |
iris_v3_frame_mover:0 |
supervisor | ok | iris_v3_frame_mover:0 RUNNING — pid 3535098, uptime 13 days, 5:36:06 | 4m ago |
iris_v3_frame_mover:1 |
supervisor | ok | iris_v3_frame_mover:1 RUNNING — pid 3535099, uptime 13 days, 5:36:06 | 4m ago |
iris_v3_frame_mover:2 |
supervisor | ok | iris_v3_frame_mover:2 RUNNING — pid 3535100, uptime 13 days, 5:36:06 | 4m ago |
other services not matched by any configured box 38 failing 23 ok
| Service | Check | Status | Message | Last seen |
|---|---|---|---|---|
categorization_ttd_asset |
supervisor | fail | categorization_ttd_asset STOPPED — Apr 30 11:06 AM | 4m ago |
google-trend-interest-enrichment |
supervisor | fail | google-trend-interest-enrichment STOPPED — Jul 14 03:12 PM | 4m ago |
imdb_extraction |
supervisor | fail | imdb_extraction FATAL — can't find command '/home/ubuntu/ttd/venv/bin/python' | 4m ago |
imdb_search |
supervisor | fail | imdb_search FATAL — can't find command '/home/ubuntu/ttd/venv/bin/python' | 4m ago |
iris_caption_1_0:0 |
supervisor | fail | iris_caption_1_0:0 STOPPED — Not started | 4m ago |
iris_caption_1_0:1 |
supervisor | fail | iris_caption_1_0:1 STOPPED — Not started | 4m ago |
iris_caption_1_0:2 |
supervisor | fail | iris_caption_1_0:2 STOPPED — Not started | 4m ago |
iris_caption_2_0:0 |
supervisor | fail | iris_caption_2_0:0 STOPPED — Not started | 4m ago |
iris_caption_2_0:1 |
supervisor | fail | iris_caption_2_0:1 STOPPED — Not started | 4m ago |
iris_caption_2_0:2 |
supervisor | fail | iris_caption_2_0:2 STOPPED — Not started | 4m ago |
iris_categorization_1_0:0 |
supervisor | fail | iris_categorization_1_0:0 STOPPED — Not started | 4m ago |
iris_categorization_1_0:1 |
supervisor | fail | iris_categorization_1_0:1 STOPPED — Not started | 4m ago |
iris_categorization_2_0:0 |
supervisor | fail | iris_categorization_2_0:0 STOPPED — Not started | 4m ago |
iris_categorization_2_0:1 |
supervisor | fail | iris_categorization_2_0:1 STOPPED — Not started | 4m ago |
segmentation_channel |
supervisor | fail | segmentation_channel STOPPED — May 25 01:22 PM | 4m ago |
segmentation_channel_2 |
supervisor | fail | segmentation_channel_2 STOPPED — Apr 09 02:39 PM | 4m ago |
segmentation_channel_3 |
supervisor | fail | segmentation_channel_3 STOPPED — Jan 28 03:25 PM | 4m ago |
segmentation_channel_4 |
supervisor | fail | segmentation_channel_4 STOPPED — Jan 28 03:25 PM | 4m ago |
tiktok-trend-interest-enrichment |
supervisor | fail | tiktok-trend-interest-enrichment STOPPED — Jul 14 03:12 PM | 4m ago |
trend-final-clustering |
supervisor | fail | trend-final-clustering STOPPED — Jul 14 03:11 PM | 4m ago |
trend-google-clustering |
supervisor | fail | trend-google-clustering STOPPED — Jul 14 03:09 PM | 4m ago |
trend-tiktok-clustering |
supervisor | fail | trend-tiktok-clustering STOPPED — Jul 14 03:11 PM | 4m ago |
trend-unified-clustering |
supervisor | fail | trend-unified-clustering STOPPED — Jul 14 03:11 PM | 4m ago |
trend-unified-interest-enrichment |
supervisor | fail | trend-unified-interest-enrichment STOPPED — Jul 14 03:12 PM | 4m ago |
ttd_categorize_1 |
supervisor | fail | ttd_categorize_1 FATAL — can't find command '/home/ubuntu/ttd/venv/bin/python' | 4m ago |
ttd_categorize_2 |
supervisor | fail | ttd_categorize_2 FATAL — can't find command '/home/ubuntu/ttd/venv/bin/python' | 4m ago |
ttd_identification_matching_data |
supervisor | fail | ttd_identification_matching_data FATAL — Exited too quickly (process log may have details) | 4m ago |
ttd_identification_preprocessor |
supervisor | fail | ttd_identification_preprocessor FATAL — Exited too quickly (process log may have details) | 4m ago |
ttd_identification_service |
supervisor | fail | ttd_identification_service STOPPED — May 29 11:23 AM | 4m ago |
ttd_iris_categorization_worker |
supervisor | fail | ttd_iris_categorization_worker STOPPED — May 11 02:00 PM | 4m ago |
ttd_iris_feeder_service |
supervisor | fail | ttd_iris_feeder_service STOPPED — Jun 22 10:16 AM | 4m ago |
ttd_iris_source_data_categorization |
supervisor | fail | ttd_iris_source_data_categorization STOPPED — Jun 22 10:16 AM | 4m ago |
ttd_iris_source_data_categorization_2 |
supervisor | fail | ttd_iris_source_data_categorization_2 STOPPED — Jun 22 10:16 AM | 4m ago |
ttd_iris_source_data_categorization_feeder |
supervisor | fail | ttd_iris_source_data_categorization_feeder STOPPED — Jun 22 10:16 AM | 4m ago |
ttd_iris_source_data_categorization_supplier |
supervisor | fail | ttd_iris_source_data_categorization_supplier STOPPED — Jun 22 10:16 AM | 4m ago |
ttd_iris_supplier_service |
supervisor | fail | ttd_iris_supplier_service STOPPED — Jun 22 10:16 AM | 4m ago |
ttd_preprocess |
supervisor | fail | ttd_preprocess FATAL — can't find command '/home/ubuntu/ttd/venv/bin/python' | 4m ago |
ttd_supply |
supervisor | fail | ttd_supply FATAL — can't find command '/home/ubuntu/ttd/venv/bin/python' | 4m ago |
categorization_feeder_ttd_asset |
supervisor | ok | categorization_feeder_ttd_asset RUNNING — pid 3275690, uptime 62 days, 22:38:07 | 4m ago |
categorization_supplier_ttd_asset |
supervisor | ok | categorization_supplier_ttd_asset RUNNING — pid 1792792, uptime 25 days, 0:31:14 | 4m ago |
channel_pipeline |
supervisor | ok | channel_pipeline RUNNING — pid 639615, uptime 96 days, 1:28:51 | 4m ago |
es_mapper_api |
supervisor | ok | es_mapper_api RUNNING — pid 3093676, uptime 4 days, 22:30:21 | 4m ago |
es_mapper_worker_gpu_v1 |
supervisor | ok | es_mapper_worker_gpu_v1 RUNNING — pid 3094348, uptime 4 days, 22:30:11 | 4m ago |
es_mapper_worker_search_v1 |
supervisor | ok | es_mapper_worker_search_v1 RUNNING — pid 3094884, uptime 4 days, 22:30:01 | 4m ago |
es_mapper_worker_search_v2 |
supervisor | ok | es_mapper_worker_search_v2 RUNNING — pid 3095112, uptime 4 days, 22:29:58 | 4m ago |
es_mapper_worker_search_v3 |
supervisor | ok | es_mapper_worker_search_v3 RUNNING — pid 3095577, uptime 4 days, 22:29:52 | 4m ago |
gemma4_feeder_service |
supervisor | ok | gemma4_feeder_service RUNNING — pid 1390400, uptime 7 days, 0:35:19 | 4m ago |
gemma4_supplier_service |
supervisor | ok | gemma4_supplier_service RUNNING — pid 1390423, uptime 7 days, 0:35:18 | 4m ago |
imdb_categorization |
supervisor | ok | imdb_categorization RUNNING — pid 4094638, uptime 6 days, 0:18:40 | 4m ago |
imdb_categorization_feeder |
supervisor | ok | imdb_categorization_feeder RUNNING — pid 4011747, uptime 6 days, 0:32:13 | 4m ago |
imdb_categorization_supplier |
supervisor | ok | imdb_categorization_supplier RUNNING — pid 817300, uptime 7:46:16 | 4m ago |
imdb_data_builder |
supervisor | ok | imdb_data_builder RUNNING — pid 2881, uptime 168 days, 5:16:12 | 4m ago |
mirros-mcp |
supervisor | ok | mirros-mcp RUNNING — pid 3404547, uptime 1 day, 1:54:35 | 4m ago |
prod_fetch_invalid_placement_ids |
supervisor | ok | prod_fetch_invalid_placement_ids RUNNING — pid 718279, uptime 76 days, 23:54:30 | 4m ago |
staging_fetch_invalid_placement_ids |
supervisor | ok | staging_fetch_invalid_placement_ids RUNNING — pid 286765, uptime 98 days, 3:01:52 | 4m ago |
ttd_identification_supplier |
supervisor | ok | ttd_identification_supplier RUNNING — pid 3435582, uptime 46 days, 4:27:17 | 4m ago |
ttd_source_data_categorization |
supervisor | ok | ttd_source_data_categorization RUNNING — pid 4095282, uptime 6 days, 0:18:35 | 4m ago |
ttd_source_data_categorization_feeder |
supervisor | ok | ttd_source_data_categorization_feeder RUNNING — pid 3198569, uptime 1 day, 4:06:32 | 4m ago |
ttd_source_data_categorization_supplier |
supervisor | ok | ttd_source_data_categorization_supplier RUNNING — pid 1516401, uptime 0:44:34 | 4m ago |
video_topic_matcher |
supervisor | ok | video_topic_matcher RUNNING — pid 2527421, uptime 1 day, 12:16:14 | 4m ago |
worker-iris_v3_captioning-6a560c6ba526df0bfa364488 |
docker | ok | container running | 4m ago |
ttd_identification TTD identification + code + backend 4 failing 2 ok
| Service | Check | Status | Message | Last seen |
|---|---|---|---|---|
ttd_iris_identification_feeder |
supervisor | fail | ttd_iris_identification_feeder STOPPED — Jun 22 10:17 AM | 4m ago |
ttd_iris_identification_matching_data |
supervisor | fail | ttd_iris_identification_matching_data STOPPED — Jun 22 10:17 AM | 4m ago |
ttd_iris_identification_service |
supervisor | fail | ttd_iris_identification_service STOPPED — May 14 12:13 PM | 4m ago |
ttd_iris_identification_supplier |
supervisor | fail | ttd_iris_identification_supplier STOPPED — Jun 22 10:17 AM | 4m ago |
ttd-identification-worker-image-anchor |
docker | ok | container running | 4m ago |
ttd-worker-image-anchor |
docker | ok | container running | 4m ago |
campaign_auditing Campaign auditing service 1 ok
| Service | Check | Status | Message | Last seen |
|---|---|---|---|---|
campaign-auditing-worker-anchor |
docker | ok | container running | 4m ago |
gpu_lb GPU load-balancer stack (workers, orchestration, routing, gemma classifier) 10 ok
| Service | Check | Status | Message | Last seen |
|---|---|---|---|---|
gemma4-classification-worker-anchor |
docker | ok | container running | 4m ago |
gpu-resource-manager-service |
docker | ok | container running | 4m ago |
health-monitoring-service |
docker | ok | container running | 4m ago |
mongodb-mongodb-1 |
docker | ok | container running | 4m ago |
orchestration-controller-service |
docker | ok | container running | 4m ago |
queue-routing-campaign-auditing |
docker | ok | container running | 4m ago |
queue-routing-service |
docker | ok | container running | 4m ago |
redis_server |
docker | ok | container running | 4m ago |
worker-destroy-service |
docker | ok | container running | 4m ago |
worker-spawn-service |
docker | ok | container running | 4m ago |
GPUs (8 · some failing)
GPU 0
_gpu_0
util100.0%
mem10819.0 / 16376.0 MB
temp91.0°C
gpu0 util=100% mem=66% temp=91°C
GPU 1
_gpu_1
util99.0%
mem15482.0 / 16376.0 MB
temp85.0°C
gpu1 util=99% mem=94% temp=85°C
GPU 2
_gpu_2
util71.0%
mem15482.0 / 16376.0 MB
temp84.0°C
gpu2 util=71% mem=94% temp=84°C
GPU 3
_gpu_3
util100.0%
mem44005.0 / 49140.0 MB
temp79.0°C
gpu3 util=100% mem=90% temp=79°C
GPU 4
_gpu_4
util78.0%
mem15482.0 / 16376.0 MB
temp84.0°C
gpu4 util=78% mem=94% temp=84°C
GPU 5
_gpu_5
util0.0%
mem828.0 / 16376.0 MB
temp43.0°C
gpu5 util=0% mem=5% temp=43°C
GPU 6
_gpu_6
util0.0%
mem10280.0 / 16376.0 MB
temp77.0°C
gpu6 util=0% mem=63% temp=77°C
GPU 7
_gpu_7
util0.0%
mem10852.0 / 16376.0 MB
temp63.0°C
gpu7 util=0% mem=66% temp=63°C