Data Pipeline

[15:40:01] INFO  Kafka consumer connected to broker-1:9092
[15:40:01] INFO  Subscribed to topics: camera.analytics.zone-*
[15:40:02] INFO  Processing batch: 1,250 events from 48 cameras
[15:40:03] INFO  Enrichment complete, writing to PostgreSQL
[15:40:03] INFO  Batch committed. Total processed: 8,945,200

[15:42] INFO  Pipeline started
[15:42] INFO  Reading from source
[15:42] INFO  Processing records
[15:42] INFO  Writing to destination
[15:42] INFO  Pipeline completed successfully

[15:40] INFO  Pipeline started
[15:40] INFO  Reading from source
[15:40] INFO  Processing records
[15:40] INFO  Writing to destination
[15:40] INFO  Pipeline completed successfully

[15:00] INFO  Pipeline started
[15:00] INFO  Reading from source
[15:00] INFO  Processing records
[15:00] INFO  Writing to destination
[15:00] INFO  Pipeline completed successfully

[00:15] INFO  Pipeline started
[00:15] INFO  Reading from source
[00:15] INFO  Processing records
[00:15] INFO  Writing to destination
[00:15] INFO  Pipeline completed successfully

[15:05:00] INFO  Starting heatmap computation for zones Z01-Z06
[15:05:15] INFO  Loading camera analytics from 14:00-15:00
[15:05:45] WARN  Zone Z01 data incomplete: 3 cameras offline
[15:06:30] ERROR OutOfMemoryError: Java heap space exceeded
[15:07:10] ERROR Pipeline failed. Scheduling retry at 15:35

[14:30] INFO  Pipeline started
[14:30] INFO  Reading from source
[14:30] INFO  Processing records
[14:30] INFO  Writing to destination
[14:30] INFO  Pipeline completed successfully

[23:00] INFO  Pipeline started
[23:00] INFO  Reading from source
[23:00] INFO  Processing records
[23:00] INFO  Writing to destination
[23:00] INFO  Pipeline completed successfully